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Abstract —  Two  ASs  are  connected  in  the  Internet  AS 
graph  only  if  they  have  a  business  “peering  relationship.” 
By  focusing  on  the  AS  subgraph  AS  pc  whose  links  repre¬ 
sent  provider-customer  relationships,  we  present  an  empirical 
study  that  identifies  three  crucial  causal  forces  at  work  in  the 
design  of  AS  connectivity:  (i)  AS-geography,  i.e.,  locality  and 
number  of  PoPs  within  individual  ASs;  (ii)  AS-specific  busi¬ 
ness  models,  abstract  toy  models  that  describe  how  individ¬ 
ual  ASs  choose  their  “best”  provider;  and  (in)  AS'  evolution,  a 
historic  account  of  the  “lives”  of  individual  ASs  in  a  dynamic 
ISP  market.  Based  on  these  findings  that  directly  relate  to 
how  provider-customer  relationships  may  be  determined  in 
the  actual  Internet,  we  develop  a  new  optimization-driven 
model  for  Internet  growth  at  the  AS  pc  level.  Its  defining 
feature  is  an  explicit  construction  of  a  novel  class  of  intuitive, 
multi-objective,  local  optimizations  by  which  the  different 
ASs  determine  in  a  fully  distributed  and  decentralized  fash¬ 
ion  their  “best”  upstream  provider.  We  show  that  our  model 
is  broadly  robust,  perforce  yields  graphs  that  match  inferred 
AS  connectivity  with  respect  to  many  different  metrics,  and 
is  ideal  for  exploring  the  impact  of  new  peering  incentives  or 
policies  on  AS-level  connectivity. 

I.  Introduction 

Internet  connectivity  at  the  level  of  Autonomous  Sys¬ 
tems  (ASs)  reflects  existing  business  relationships  among 
ASs.  Two  ASs  are  connected  in  an  AS  graph  by  a  link  only 
if  they  have  a  “peering  relationship”  between  them,  e.g., 
provider-customer  or  peer-to-peer  relationship.  In  princi¬ 
ple,  snapshots  of  the  Internet’s  AS  graph  can  be  inferred 
from  BGP-derived  measurements,  but  in  practice,  the  re¬ 
sulting  graph  structures  require  careful  interpretation.  For 
example,  since  the  measurements  consist  of  a  collection 
of  snapshots  of  BGP  routing  tables  taken  at  a  few  van¬ 
tage  points  on  the  Internet  over  time,  private  peering  links 
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and  backup  connections  between  ASs  generally  cannot  be 
identified,  because  they  remain  largely  invisible  to  collec¬ 
tion  sites  such  as  the  Oregon  route  server  [28].  Thus,  the 
resulting  inferred  AS  graphs  tend  to  be  less  densely  con¬ 
nected  than  the  actual  Internet  at  the  AS  level  [12],  and 
the  inferred  peering  relationships  are  not  always  accurate. 
These  difficulties  not  withstanding,  a  striking  characteris¬ 
tic  of  various  inferred  AS  graphs  (with  more  or  less  in¬ 
complete  connectivity  information)  has  been  the  observed 
high  variability  of  the  AS  vertex  degrees,  parsimoniously 
captured  by  vertex  degree  distributions  of  the  power-law 
type  [15],  [12],  This  power-law  finding  has  led  to  renewed 
interest  in  network  modeling  and  has  motivated  the  de¬ 
velopment  of  new  topology  generators.  However,  much 
of  the  efforts  to  date  has  been  either  abstract  (e.g.,  gen¬ 
erate  a  graph  with  a  given  vertex  degree  distribution  [1], 
[21])  or  based  on  some  exogenously  imposed  mechanisms 
(e.g.,  a  presumed  preferential-type  connectivity  rule  [2], 
[24]).  While  the  resulting  models  and  generators  are  gen¬ 
erally  successful  in  reproducing  and  matching  the  power- 
law  type  node  degree  distributions  of  measured  AS  graphs, 
their  relevance  to  networking  is  seriously  hampered  by 
their  generic  nature — they  are  mostly  designed  to  model 
all  types  of  networks  that  show  power-law  type  node  de¬ 
gree  distributions  [7].  As  a  result,  these  models  completely 
ignore  any  AS-specific  factors  and  criteria  inherent  in  es¬ 
tablishing  the  very  business  relationships  expressed  by  AS 
graphs.  The  models’  theoretical  appeal  is  also  limited, 
mainly  because  of  the  models’  exclusive  focus  on  a  single 
metric  (i.e.,  node  degree  distribution)  and  a  general  inabil¬ 
ity  to  also  match  in  a  parsimonious  manner  inferred  AS 
graphs  with  respect  to  alternative  metrics  (e.g.,  hierarchy- 
related  or  graph  evolution-specific  measures). 

The  main  objective  of  this  paper  is  to  explore  a  radi¬ 
cally  different  approach  to  modeling  and  generating  In¬ 
ternet  topologies  at  the  AS  level. 1  Instead  of  relying  on 

1 A  similar  approach  has  recently  been  advocated  and  outlined  in  [3] 
for  modeling  Internet  connectivity  at  the  router-level,  but  no  results 
have  been  reported  to  date. 
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abstract  or  artificially  imposed  mechanisms  that  yield  AS 
graphs  which  manage  to  attain  highly  variable  node  degree 
distributions  but  fail  to  match  other  graph  characteristics, 
we  focus  here  on  identifying  key  causal  forces  at  work  in 
the  fully  distributed  and  decentralized  design  of  AS  inter¬ 
connectivity.  More  specifically,  we  are  mainly  concerned 
with  the  subgraph  ASpc  of  the  overall  AS  graph  whose 
links  represent  only  provider-customer  peering  relation¬ 
ships,  i.e.,  non  provider-customer  relationships  arc  not  paid 
of  this  subgraph.  We  show  that  by  explicitly  incorporat¬ 
ing  provider-customer  business  relationships  (or  concrete 
abstractions  thereof)  into  an  appropriate  network  growth 
model,  the  resulting  AS  subgraphs  automatically  match  in¬ 
ferred  A.S'pc'-specific  connectivity  with  respect  to  a  wide 
range  of  metrics.  Furthermore,  these  subgraphs  share  a 
number  of  characteristics  with  the  overall  AS  graphs  (e.g., 
highly  variable  node  degree  distributions).  This  finding 
demonstrates  the  importance  of  the  ASpc  subgraphs  in 
gaining  a  better  understanding  of  the  Internet’s  overall  AS 
connectivity  and  it  argues  for  a  detailed  study  of  the  prop¬ 
erties  and  dynamic  nature  of  these  AS1  pc  subgraphs. 

Our  overall  approach  is  motivated  by  the  recently  pro¬ 
posed  HOT  ( Highly  Optimized  Tolerance)  concept  intro¬ 
duced  by  Carlson  and  Doyle  [11],  [13].  HOT  provides  a 
general  framework  in  which  highly  variable  event  sizes,  in 
systems  highly  optimized  by  engineering  design,  are  the 
result  of  tradeoffs  between  yield,  cost  of  resources,  and 
the  systems’  tolerance  to  risk.  In  turn,  the  HOT  frame¬ 
work  emphasizes  the  importance  of  design,  structure,  and 
optimization  in  the  study  of  highly  engineered  and  com¬ 
plex  systems,  such  as  the  Internet.  In  applying  this  general 
HOT  concept  to  the  specific  problem  addressed  in  this  pa¬ 
per,  our  work  draws  heavily  on  the  first  explicit  attempt 
by  Fabrikant  el  al.  [14]  to  cast  network  design  as  a  HOT 
problem.2  In  Fabrikant  et  aids  generic  toy  model  of  net¬ 
work  growth,  each  newly  arriving  node  decides  on  the  par¬ 
ticular  node  of  the  existing  graph  it  connects  to  by  solv¬ 
ing  a  generic  multi-objective  optimization  problem.  This 
model  serves  as  starting  point  of  our  investigation  into  the 
causal  forces  shaping  Internet  connectivity  at  the  ASpc 
level.  That  is,  following  Fabrikant  et  aids  model,  we  at¬ 
tempt  to  determine  a  particular  class  of  multi-objective  op¬ 
timization  problems  that  reflect  the  various  factors  and  cri¬ 
teria  by  which  provider  and  customer  ASs  determine  their 
peering  relationships  with  one  another. 

Our  proposed  approach  to  Internet  modeling  at  the 
AS  pc  level  has  three  major  novel  components.  First, 
in  contrast  to  the  generic  Internet  growth  model  of  Fab¬ 
rikant  et  al.  that  deals  with  generic  nodes  and  links,  we 

2  Fabrikant  et  al.  also  suggested  Heuristically  Optimized  Tradeoff  as 
a  fi  tting  alternative  acronym  for  HOT. 


provide  the  first  explicit  attempt  to  cast  the  design  of  AS 
level  connectivity  as  a  HOT  problem.  In  particular,  we 
develop  a  new  HOT  model  for  AS pc  graphs  that  fully  ex¬ 
ploits  the  networking  semantics  of  the  relevant  objects — 
the  graph's  nodes  represent  businesses  (i.e.,  provider  or 
customer  ISPs)  and  its  links  express  explicit  business  re¬ 
lationships  among  these  nodes.  Second,  we  formulate  a 
novel  class  of  multi-objective  optimization  problems  to 
capture  the  decision  process  ASs  use  to  determine  their 
peering  or  business  relationships  with  other  ASs.  To  ar¬ 
rive  at  this  formulation,  we  proceed  in  three  steps  and 
show  successively  that  (i)  AS  geography,  (ii)  AS  business 
model,  and  (iii)  AS  evolution  define  three  key  criteria  that 
must  be  accounted  for.  We  propose  concrete  abstractions 
of  each  of  these  criteria  and  incorporate  them  one-by-one 
into  our  multi-objective  optimization  problems.  We  illus¬ 
trate  how  these  optimization  problems  shape  the  forma¬ 
tion  of  local  provider-customer  relationships  and  cause  the 
“emerging”  global  characteristics  exhibited  by  the  result¬ 
ing  AS  pc  graphs.  In  addition,  we  show  that  a  number  of 
features  associated  with  the  overall  AS  graphs  (e.g.,  high 
variability  in  node  degree  distributions)  arc  already  present 
in  the  corresponding  ASpc  subgraphs  and  can  therefore 
be  explained  and  understood  in  terms  of  the  causal  forces 
at  work  in  the  design  of  the  ASpc  portion  of  the  overall 
AS  graph.  Finally,  we  illustrate  along  the  way  that  the  pro¬ 
posed  HOT  model  for  Internet  growth  at  the  ASpc  level 
has  a  number  of  attractive  robustness  properties  with  re¬ 
spect  to  the  details  of  how  the  three  identified  key  criteria 
are  abstracted  and  expressed  as  objectives  that  have  to  be 
optimized  simultaneously.  These  insensitivity  results  en¬ 
hance  the  overall  credibility  of  the  proposed  HOT  model. 
They  also  make  the  resulting  model  especially  appealing 
for  exploring  a  range  of  what-if  scenarios.  For  example,  if 
modifications  to  existing  criteria  for  establishing  provider- 
customer  relationships,  or  the  introduction  of  new  ones, 
may  more  accurately  reflect  incentives  in  future  provider- 
customer  relationships,  our  model  can  be  used  to  explore 
the  impact  such  modifications  or  embellishments  may  have 
on  the  overall  AS  connectivity. 

Due  to  the  generic  nature  of  previous  efforts  to  modeling 
Internet  topologies,  hitherto  none  of  the  three  key  criteria 
we  propose  has  played  any  significant  role  in  modeling  the 
AS  graphs.  For  the  non-generic,  networking-centric  ap¬ 
proach  we  propose,  the  importance  of  these  three  criteria 
should  come  as  no  surprise.  Consider  for  example  “AS  ge¬ 
ography,”  by  which  we  mean  the  number  and  geographic 
locations  of  an  AS’s  Points-of-Presence  (PoPs).  Clearly, 
knowing  the  geography  of  intra-AS  PoP  structures  is  im¬ 
portant  to,  say,  distinguish  between  nearby  and  far-away 
providers — with  obvious  implications  in  determining  con- 
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nection  costs.  Similarly,  depending  on  an  AS’s  business 
model  (which  may  include  technological  aspects  such  as 
network  availability,  network  reliability,  expected  perfor¬ 
mance,  or  support  for  value-added  services,  as  well  as 
economic  considerations  such  as  pricing  plans,  projected 
network  build-out,  customer  support,  etc.),  customer  ASs 
can  be  expected  to  behave  in  a  more  or  less  rational  man¬ 
ner  when  establishing  upstream  connectivity;  for  exam¬ 
ple,  they  might  choose  among  competing  providers  by 
implicitly  or  explicitly  satisfying  some  local  Pareto  op¬ 
timality  criterion  with  respect  to  some  underlying  utility 
measure.  As  for  AS  evolution,  the  dynamics  of  the  ISP 
market  (overall  growth  amidst  ever-present  mergers,  ac¬ 
quisitions,  and  bankruptcies)  cause  or  force  ASs  to  pe¬ 
riodically  re-examine  their  peering  relationships  and,  if 
necessary,  establish  new,  get  rid  of  old,  or  modify  exist¬ 
ing  relationships — with  obvious  implications  on  the  over¬ 
all  connectivity  at  the  AS  level. 

We  hasten  to  point  out  that  while  the  generation  of  re¬ 
alistic  AS  topologies  is  also  a  natural  by-product  of  our 
approach,  it  is  not  the  focus  of  this  paper.  In  particular,  to 
extend  our  approach  and  develop  a  HOT  model  for  Inter¬ 
net  growth  at  the  overall  AS  level  will  require  incorporat¬ 
ing  peer-to-peer  relationships  and  accounting  for  ASs  that 
change  from  peer  to  provider/customer  or  vice  versa.  In 
this  sense,  the  causal  forces  identified  in  this  paper  and  ac¬ 
counted  for  in  the  proposed  HOT  model  arc  necessary  but 
not  sufficient  for  shaping  the  existing  Internet  topology  at 
the  overall  AS  level.  For  example,  while  our  model  suc¬ 
ceeds  in  explaining  the  power-law  type  node  degree  distri¬ 
butions  of  inferred  AS  graphs  in  terms  of  AS  geography- 
related  characteristics  (i.e.,  AS  size  measured  in  terms  of 
number  of  PoPs  per  AS),  it  also  suggests  the  presence  of 
additional  causal  forces  that  relate  directly  to  the  peer-to- 
peer  portion  of  AS  graphs.  These  forces  will  require  care¬ 
ful  consideration  when  extending  our  approach  and  devel¬ 
oping  a  more  complete  model  for  Internet  connectivity  at 
the  overall  AS  level.  However,  we  leave  the  pursuit  of  such 
a  modeling  effort  for  future  research. 

The  rest  of  the  paper  is  structured  as  follows.  In  Sec¬ 
tion  II,  we  describe  the  two  inferred  AS  graphs  that  form 
the  basis  of  our  empirical  study  and  reduce  them  to  simpler 
tree  topologies  that  are  used  in  the  subsequent  sections. 
We  discuss  in  Section  III  the  HOT  model  for  generic  In¬ 
ternet  growth  proposed  in  [14]  and  demonstrate  its  short¬ 
comings  when  applied  to  the  AS  graph.  The  construction 
of  our  HOT  model  for  Internet  growth  at  the  AS  pc  level 
is  described  in  Sections  IV-VI,  where  we  identify,  in  suc¬ 
cession,  AS  geography,  AS  business  model,  and  AS  evo¬ 
lution  as  key  forces  at  work  in  the  design  of  Internet  AS 
connectivity.  We  conclude  the  paper  with  a  discussion  of 


TABLE  I 

AS  RELATIONSHIP  INFERENCE  RESULT 


AS  graph 

Number  of  links 

All 

Provider-customer 

Peer-to-peer 

Others 

Oregon 

23,449 

21,473  (91.6%) 

1,621  (6.9%) 

355 

Oregon+ 

32,759 

27,815  (84.9%) 

3,919(12.0%) 

1,025 

some  practical  implication  of  our  proposed  approach  and 
the  ensuing  models  and  comment  on  the  work’s  impact  on 
network  modeling  as  a  scientific  discipline. 

II.  Inference  for  AS  graphs 

We  address  in  this  section  some  of  the  challenges  re¬ 
lated  to  inferring  AS  graphs  and  describe  our  approach  for 
tackling  them  in  ways  that  are  necessarily  imperfect,  but 
still  ensure  the  validity  of  our  results. 

A.  Inferring  AS  Connectivity 

In  principle,  to  determine  whether  AS  X  and  AS  Y  have 
a  peering  relationship  with  one  another,  all  that  is  needed 
is  to  collect  X’s  BGP  routing  table  and  check  if  AS  Y 
appeal's  in  any  of  the  table’s  AS-path  entries.  In  prac¬ 
tice,  however,  only  a  very  limited  number  of  ASs  make 
their  BGP  routing  tables  publicly  available.  As  a  result,  all 
available  inferred  AS  graphs  are  necessarily  incomplete. 
While  this  incompleteness  should  always  be  kept  in  mind 
when  reporting  observed  characteristics  of  measured  AS 
graphs,  it  does  not  necessarily  invalidate  all  of  the  find¬ 
ings.  A  particularly  attractive  method  to  demonstrate  the 
general  validity  of  a  given  AS -related  observation  despite 
the  incompleteness  property  is  to  illustrate  that  the  result 
in  question  is  robust  with  respect  to  alternative  AS  graph 
inference  techniques  or  to  reliance  on  additional  relevant 
data  (e.g.,  more  BGP  routing  tables,  Looking  Glass  infor¬ 
mation,  etc.). 

To  explore  the  robustness  of  the  findings  reported  in 
this  paper,  we  rely  on  two  inferred  AS  graphs  that  have 
been  studied  in  detail  in  [12],  Most  importantly,  these  AS 
graphs  have  significantly  different  connectivity  densities. 
One  graph,  the  OREGON  AS  graph,  is  constructed  exclu¬ 
sively  from  information  contained  in  the  BGP  routing  ta¬ 
bles  collected  by  the  Oregon  route  server  [28],  This  AS 
graph  has  been  the  most  widely-used  graph  in  past  studies 
of  the  Internet’s  AS  topology.  The  one  used  here  is  based 
on  data  collected  in  late  May  of  2001.  The  second  graph, 
the  OREGON+  AS  graph,  relies  not  only  on  the  Oregon 
BGP  data,  but  also  makes  use  of  a  number  of  additional 
BGP  routing  tables,  BGP  summary  information  obtained 
by  querying  numerous  ASs  using  the  Looking  Glass  tool, 
and  diligently  extracted  data  from  the  Routing  Registry 
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(a)  Oregon  topology 


(b)  Oregon+  topology 


Fig.  1.  Node  degree  distributions  of  AS  (sub)graphs. 


TABLE  II 

AS  GRAPH  STATISTICS 


AS  graph 

All 

prov-cust  subgraph 

peer-to-peer  subgraph 

All 

Largest  comp. 

All 

Largest  comp. 

Oregon 

#  of  nodes 

11.183 

11,177 

11,177  (100%) 

512 

512(100%) 

#  of  links 

23,449 

21,473 

21,473  (100%) 

1,621 

1,621  (100%) 

Oregon+ 

#  of  nodes 

11.456 

11,257 

11,257  (100%) 

1,365 

837  (61.3%) 

#  of  links 

32,759 

27,815 

27,815  (100%) 

3,919 

3,359  (85.7%) 

database.3  The  data  collection  period  is  again  late  May 
2001.  While  this  graph  has  about  the  same  number  of  ASs 
as  the  OREGON  AS  graph.  Table  II  (first  column)  shows 
that  it  has  about  40%  more  links  than  the  OREGON  AS 
graph.  Realizing  that  both  of  these  AS  graphs  exhibit  qual¬ 
itatively  the  same  power-law  type  node  degree  distribution 
(e.g.,  see  Fig.  1,  lines  labeled  “all”)  suggests  that  this  fea¬ 
ture  is  robust  with  respect  to  the  degree  of  incompleteness 
of  the  two  graphs  and  can  therefore  be  considered  a  gen¬ 
uine  characteristic  of  Internet  connectivity  at  the  AS  level. 
It  also  justifies  our  pursuit  of  a  careful  investigation  into 
the  underlying  causes  of  this  striking  characteristic. 

B.  Provider-customer  v.v.  Peer-to-peer  Relationships 

We  are  first  interested  in  whether  the  power-law  type 
node  degree  distributions  observed  in  both  of  our  AS 
graphs  arc  associated  with  any  specific  peering  type,  in 
which  case  we  may  simplify  our  study  by  removing  the 
non-causal  link  types  from  the  graphs.  A  link  between  two 
ASs  in  an  AS  graph  typically  reflects  either  a  provider- 
customer  or  a  peer-to-peer  relationship.4  In  the  former, 
one  AS  plays  the  role  of  the  customer,  while  the  other  is 
the  provider  of  Internet  connectivity.  Internet  providers 

3  The  use  of  selected  information  from  the  Routing  Registry  database 
is  discussed  and  justifi  ed  in  [12]  and  is  not  essential  for  the  purposes  of 
this  paper. 

4Other  relationships  are  possible,  for  example  as  backup  links  be¬ 

tween  non-provider  ASs,  but  are  relatively  rare.  They  are  grouped  un¬ 

der  ‘Others”  in  Table  I. 


are  paid  by  their  customers  for  providing  this  service.  In 
the  latter,  the  ASs  see  equal  benefit  in  interconnecting 
with  each  other  and  no  financial  exchange  takes  place. 
Given  the  BGP  routing  tables  (plus  other  information,  if 
available)  used  to  infer  an  AS  graph,  we  can  annotate  the 
links  in  the  graph  with  inferred  peering  relationships  us¬ 
ing  heuristics  proposed  in  [17]  and  [30]. 5  The  results  of 
augmenting  the  OREGON  and  OREGON+  AS  graphs  with 
inferred  peering  relationships  arc  summarized  in  Table  I. 

To  determine  whether  the  power-law  type  node  degree 
distributions  observed  in  our  inferred  AS  graphs  are  as¬ 
sociated  with  any  specific  peering  type,  we  consider  two 
separate  subgraphs  for  each  of  the  Oregon  and  Ore- 
GON+  AS  graphs.  The  provider-customer  (or  prov-cust, 
for  short)  subgraph  contains  only  provider-customer  links 
(along  with  their  incident  ASs),  while  the  peer-to-peer 
subgraph  consists  of  peer-to-peer  links  only  (and  all  their 
incident  ASs).6  Table  II  provides  details  on  these  sub¬ 
graphs  (or  their  largest  connected  components). 

Fig.  1  compares  the  node  degree  distributions  of  the 
original  OREGON  AS  graph  (line  labeled  “all”)  and  its  two 
subgraphs  (lines  labeled  “prov-cust  subgraph”  and  “peer- 
to-peer  subgraph”)  in  plot  (a),  and  the  original  OREGON+ 

5We  need  to  use  both  heuristics  due  to  limitations  of  each  one  of 
them.  For  example,  the  heuristic  proposed  in  [17]  cannot  be  used  on 
our  OREGON+  graph  that  contains  links  inferred  from  Looking  Glass 
data  but  are  not  present  in  any  of  the  collected  BGP  tables. 

6In  case  one  of  the  resulting  subgraphs  is  not  connected,  we  consider 
its  largest  connected  component. 
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(a)  Oregon  topology  (b)  Oregon+  topology 

Fig.  2.  Node  degree  distributions  of  PI  subgraphs. 


AS  graph  and  its  two  subgraphs  in  plot  (b).  As  can  be 
clearly  seen,  in  both  cases  the  node  degree  distributions 
of  the  provider-customer  subgraphs  arc  almost  identical 
to  those  of  the  original  AS  graphs,  but  those  correspond¬ 
ing  to  the  peer-to-peer  subgraphs  behave  qualitatively  dif¬ 
ferent.  We  can  conclude  that  peer-to-peer  and  other  non 
provider-customer  links  arc  non-causal  factors  as  far  as  the 
power-law  type  node  degree  distribution  characteristic  of 
AS  graphs  is  concerned.  In  turn.  Fig.  1  suggests  that  it 
may  suffice  to  focus  mainly  on  the  provider-customer  por¬ 
tion  of  the  AS  graph  when  attempting  to  identify  some  of 
the  causal  forces  at  work  in  the  design  of  Internet  connec¬ 
tivity  at  the  AS  level. 

C.  Single-  vs.  Multi-homing 

To  further  simplify  our  study  of  AS  graph  structures, 
we  next  ask  if  multi-homing  has  any  major  impact  on  the 
power-law  type  node  degree  distribution.7  Multi-homing 
typically  refers  to  a  customer  AS  having  more  than  one 
peering  link  with  either  the  same  provider  AS  or  with 
different  provider  ASs.  The  purpose  of  multi-homing  is 
mainly  to  improve  an  AS’s  reliability  or  performance  (e.g., 
via  load-balancing).  To  gauge  the  effect  of  multi-homing 
on  AS -connectivity,  we  consider  three  heuristics  for  dis¬ 
tinguishing  between  the  primary  connection  and  the  sec¬ 
ondary  connection(s)  of  an  AS.8  For  the  first  heuristic, 
given  an  AS  X,  we  define  its  best  provider  (among  its 
current  providers)  to  be  the  AS  with  the  lowest  average 
hop  distance  to  all  the  other  ASs  in  the  provider-customer 
subgraph.  The  second  heuristic  redefines  the  best  provider 
of  AS  X  to  be  the  one  with  the  largest  AS  node  degree 

'  The  increase  in  multi-homed  networks  has  been  a  driving  force  be¬ 
hind  the  recent  growth  in  the  size  of  BGP  routing  tables  [20]. 

8 Multi-homing  for  the  purpose  of  load-balancing  may  consider  all  the 
multi-homing  links  as  equally  important.  However,  in  this  paper,  we 
assume  that  even  in  such  a  case,  there  still  exists  one  primary  upstream 
connection  carrying  a  major  portion  of  an  AS’s  traffi  c. 


among  all  of  X’s  current  providers,  while  the  third  heuris¬ 
tic  randomly  picks  one  of  AS  X's  providers  as  the  best 
provider.  Thus,  by  annotating  the  links  of  the  provider- 
customer  subgraph  as  primary  (link  to  best  provider)  or 
secondary ,  we  obtain  three  different  instances  of  the  sub¬ 
graph,  one  per  heuristic. 

For  each  of  these  three  annotated  provider-customer 
subgraphs,  we  generate  a  new  subgraph  (called  the  PI  sub¬ 
graph)  by  removing  all  secondary  links.  These  “single- 
homed”  PI  subgraphs  of  both  the  OREGON  and  Ore- 
GON+  provider-customer  subgraphs  have  approximately 
50%  fewer  links  than  their  “multi-homed”  counterparts, 
i.e.,  their  corresponding  provider-customer  subgraphs. 
Fig.  2  depicts  the  node  degree  distributions  of  the  in¬ 
ferred  provider-customer  subgraph  (labeled  “prov-cust 
subgraph”)  and  its  three  PI  subgraphs  (labeled  “prov-cust 
subgraph  single-homed  (hop  distance),”  “prov-cust  sub¬ 
graph  single-homed  (degree),”  and  “prov-cust  subgraph 
single-homed  (random)”)  of  the  OREGON  and  OREGON+ 
graphs  in  plots  (a)  and  (b),  respectively.  The  figure 
shows  that  the  P 1  subgraphs  exhibit  qualitatively  the  same 
node  degree  distributions  as  the  corresponding  provider- 
customer  subgraphs,  except  that  the  entire  distributions 
are  shifted  down,  with  a  slight  change  in  slopes.  Being 
largely  independent  of  the  original  AS  graph  (i.e.,  ORE¬ 
GON  vs.  OREGON+)  and  of  the  details  of  defining  a  cus¬ 
tomer’s  “best”  provider  (i.e.,  the  three  instances  of  the  Pi 
subgraph).  Fig.  2  suggests  that  multi-homing  is  not  respon¬ 
sible  for  the  power-law  type  node  degree  distributions  and 
not  a  causal  force  behind  this  property. 

The  PI  subgraphs,  in  which  every  AS  has  a  single 
provider,  form  forest  topologies,  possibly  with  multiple 
root  nodes.  However,  it  turns  out  that  for  both  the  ORE¬ 
GON  and  OREGON+  graphs,  the  largest  (connected)  tree 
contained  in  their  respective  PI  subgraphs  is  rooted  at 
AS701  (UUNET)  and  spans  about  99%  of  all  of  Pi’s 
nodes.  For  the  remainder  of  this  paper,  when  exploring 
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the  causal  forces  at  work  in  shaping  Internet  connectivity 
at  the  AS  pc  level,  we  will  focus  mainly  on  this  simplified 
AS  graph  consisting  of  the  largest  connected  tree  topology 
rooted  at  AS701. 

III.  AS  Growth — an  Optimization-Driven 
Process 

The  orthodox  physics  views  tend  to  associate  the  ubiq¬ 
uity  of  power-law  distributions  in  natural  and  engineered 
complex  systems  unambiguously  with  critical  phase  tran¬ 
sition  [6].  However,  in  the  specific  case  of  the  Internet, 
where  power-law  type  distributions  abound,  this  apparent 
connection  turns  out  to  be  specious  and  can  be  directly  re¬ 
futed  [32].  As  a  result,  the  Internet  has  become  a  prime 
target  for  testing  the  validity  of  alternative  views  and  theo¬ 
ries  to  explain  the  ubiquity  of  power-law  type  distributions 
in  nature  and  engineering.  An  especially  promising  and 
radically  different  such  alternative  view  has  recently  been 
proposed  by  Carlson  and  Doyle  [11],  [13]  and  is  based 
on  the  concept  of  HOT  (for  Highly  Optimized  Tolerance). 
In  this  section,  we  summarize  the  state-of-the-art  of  us¬ 
ing  HOT-based  approaches  to  model  Internet  growth  and 
investigate  their  relevance  for  network  design  at  the  AS 
level. 

A.  The  Generic  HOT  Model  of  Fabrikant  et  al. 

The  HOT  concept  introduced  in  [11],  [13]  emphasizes 
the  importance  of  design,  structure,  and  optimization  and 
provides  a  framework  in  which  power-law  type  event  size 
distributions  in  systems  optimized  by  engineering  design 
are  the  results  of  tradeoffs  between  yield,  cost  of  resources, 
and  tolerance  to  risk.  Highly  Optimized  alludes  to  the 
fact  that  robustness  (i.e.,  the  maintenance  of  some  de¬ 
sired  system  characteristics  despite  uncertainties  in  the 
behavior  of  its  component  parts  or  its  environment)  is 
achieved  by  highly  structured,  rare,  non-generic  config¬ 
urations  which — for  highly  engineered  systems — arc  the 
result  of  deliberate  design.  Tolerance  emphasizes  that  this 
robustness  in  complex  systems  is  a  constrained  and  limited 
quantity  and  must  be  diligently  managed. 

The  first  explicit  attempt  to  cast  network  design,  mod¬ 
eling,  and  generation  as  a  HOT  problem  was  instigated  by 
Fabrikant  et  al.  [14].  They  proposed  a  toy  model  of  In¬ 
ternet  growth  in  which  each  newly  arriving  generic  node 
establishes  connectivity  by  solving  locally  identical  types 
of  simple  multi-objective  optimization  problems.  A  new 
node  attempts  to  simultaneously  optimize  two  objectives: 
“last  mile”  connection  cost  and  “node  centrality”  cost.  The 
“last  mile”  connection  cost  is  supposed  to  capture  the  cost 
of  resources  associated  with  connecting  to  a  parent  node 
and  is  measured  in  terms  of  Euclidean  distance.  The  “node 


centrality”  cost  reflects  transmission  delays  and  is  mea¬ 
sured  in  terms  of  the  average  hop  distance  from  a  potential 
parent  to  all  other  nodes  in  the  graph.  More  formally,  as¬ 
suming  a  tree  topology,  when  a  node  i  joins  a  graph,  it 
attaches  itself  to  the  node  j  that  minimizes  the  weighted 
sum  of  the  two  objectives:  min^fa  ■  dij  +  hj ),  where 
dij  is  the  Euclidean  distance  between  i  and  j,  and  hj  is 
the  average  hop  distance  from  j  to  the  entire  graph.  De¬ 
pending  on  the  relative  importance  of  the  two  objectives 
in  this  multi-objective  optimization,  i.e.,  depending  on  the 
value  of  the  parameter  a,  the  authors  prove  in  [14]  that 
their  HOT  model  yields  three  different  regimes  of  graphs 
with  qualitatively  very  different  hierarchical  structures  and 
node  degree  distributions.  Weighting  the  transmission  de¬ 
lays  over  the  connection  cost  (small  values  of  a)  creates 
star-like  topologies,  centered  primarily  around  a  single 
node  (“hot  spot”).  If  the  connection  cost  dominates  over 
the  transmission  delays  (large  values  of  a),  the  resulting 
graphs  are  more  like  random  graphs,  with  exponential-tail 
type  node  degree  distributions.  The  interesting  regime  lies 
in  between  (i.e.,  medium  a-values)  and  consists  of  graphs 
with  power-law  type  node  degree  distributions. 

B.  The  Generic  HOT  Model  and  AS  Connectivity 

Although  never  stated  explicitly  in  [14],  the  HOT  model 
of  Fabrikant  et  al.  is  generic  in  the  sense  that  its  nodes 
are  neither  routers  nor  ASs,  and  its  links  express  neither 
router-level  physical  connectivity  nor  inter- AS  peering  re¬ 
lationships.  Since  our  focus  is  on  AS-level  connectivity, 
we  first  ask  whether  the  proposed  HOT  model  is  indeed 
appropriate  and  relevant  to  an  AS-centric  description  of 
Internet  growth. 

When  interpreting  a  graph’s  nodes  as  ASs,  the  HOT 
model  of  Fabrikant  et  al.  assumes  a  randomly-chosen  point 
location  for  each  AS  and  motivates  the  particular  multi¬ 
objective  optimization  problem  by  arguing  that  in  terms  of 
connection  cost,  there  exists  a  trade-off  between  being  at¬ 
tracted  to  the  topological  core  and  being  geographically 
separated  from  the  core.  However,  a  typical  AS  or  ISP 
maintains  in  general  multiple  Point  of  Presences  (PoPs) 
within  its  networks,  and  each  PoP  is  a  physical  access  lo¬ 
cation  where  customer  ASs  can  connect  to  (e.g.,  see  [18], 
[29]).  Naturally,  for  a  customer  AS  choosing  among  com¬ 
peting  upstream  providers,  important  considerations  in¬ 
clude:  does  the  provider  have  a  PoP  nearby,  and  how  many 
PoPs  does  the  provider  maintain  globally  [23].  Further¬ 
more,  large  ASs  close  to  the  topological  core  can  typically 
afford  to  invest  more  financial  resources  in  their  network 
infrastructure  and  by  doing  so,  tend  to  further  increase 
their  global  reach  (i.e.,  number  of  PoPs  and  geographic 
diversity  of  PoP  locations).  Consequently,  large  ASs  near 


7 


Node  Rank 

(a)  Number  of  locations  per  node 

Fig.  3.  The  modifi  ed  HOT 

the  topological  core  arc  more  likely  to  also  be  in  closer  ge¬ 
ographic  proximity  to  new  customers  than  smaller  ASs, 
which  in  addition  arc  predominately  located  around  the 
topological  edges.  This  argument  illustrates  that  when  ap¬ 
plying  the  HOT  model  of  Fabrikant  et  al.  directly  to  Inter¬ 
net  growth  at  the  AS  level,  the  two  proposed  optimization 
objectives  may  no  longer  be  independent — when  nodes  arc 
multi-PoP  ASs,  a  candidate  node  with  low  transmission 
delays  has  presumably  also  a  low  “last  mile”  connection 
cost.  As  a  consequence,  in  a  setting  that  is  more  realis¬ 
tic  and  allows  for  multi-PoP  ASs,  the  HOT  model  of  Fab¬ 
rikant  et  al.  can  be  expected  to  yield  star-shaped  graphs. 

To  test  this  hypothesis,  we  modify  the  generic  HOT 
model  of  Fabrikant  et  al.  to  account  for  multi-PoP  AS 
structures  as  follows.  Each  existing  node  u  in  the  graph 
has  a  location  list  locJist(u),  which  contains  the  set  of 
its  locations  (as  in  [14],  we  work  in  the  unit  square,  but 
as  usual,  the  shape  is  inconsequential).  For  a  new  node 
i,  locJist(i )  is  initialized  to  contain  a  single  randomly 
chosen  location.  To  attach  the  new  node  i  to  the  graph 
with  i- 1  existing  nodes  labeled  1  through  i- 1 ,  instead  of 
connecting  it  to  target  node  j  that  satisfies  the  generic 
HOT  criterion,  i.e.,  minJ<r{a  ■  dij  +  hj),  we  connect  it 
to  target  node  j  that  satisfies  the  modified  HOT  criterion: 
mirijc^a  ■  (' minmocUst^du )  +  hj).  That  is,  when  a 
newly  added  node  searches  for  a  node  to  attach  to,  it  con¬ 
siders  the  same  optimization  problem  as  for  the  generic 
HOT  model,  except  that  each  candidate  node  now  has  mul¬ 
tiple  locations.  Thus,  for  each  candidate  node,  its  closest 
location  has  to  be  found  first. 

To  reflect  our  intuition  that  with  a  growing  customer 
base,  ASs  will  increase  their  number  of  PoPs,  and  that  the 
likelihood  of  a  larger  AS  installing  an  additional  PoP  is 
greater  than  that  of  a  smaller  AS,  after  node  i  attaches  to 
the  graph,  each  existing  node  u  (nodes  1  to  i  —  1)  is  given  a 
chance  to  increment  the  number  of  its  PoPs  by  one.  Specif¬ 
ically,  with  probability  pioc{u),  each  existing  node  a  adds 
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(b)  Node  degree 
model  (a=5  and  n=  10,000) 

a  new  random  location  to  locJist(u)\  with  probability  1- 
Pioc{u),  no  new  location  is  added.  The  probability  pioc(u) 
is  given  as  K  ■  rank(u)  ^  where  K  and  (3  are  positive 
constants  and  rank(u )  is  the  rank  of  node  u  when  all  the 
existing  nodes  in  the  graph  arc  sorted  by  the  number  of 
their  children  nodes  in  a  monotonically  decreasing  order. 
The  parameter  K  constrains  the  maximum  number  of  lo¬ 
cations  per  node  and  satisfies  0  <  K  <  1;  the  maximum 
number  of  locations  per  node  increases  as  K  increases.  On 
the  other  hand,  (3  governs  the  decay  of  the  distribution  of 
the  number  of  locations  per  node  (i.e.,  exponential  type 
decay  for  small  /3-values,  power-law  type  decay  for  large 
(3- values).  We  call  this  the  modified  HOT  model  for  In¬ 
ternet  growth  at  the  AS  level.  In  contrast  to  the  generic 
model  of  Fabrikant  et  al. ,  this  modified  model  allows  for 
multi-PoP  ASs  and  attempts  to  capture  their  evolution  in 
time.  As  the  connectivity  of  the  graph  grows  over  time, 
the  internal  structure  (i.e.,  the  number  of  PoPs  and  their 
geographic  locations)  of  individual  nodes  can  also  expand. 

Fig.  3  shows  the  results  of  three  sample  graphs  gen¬ 
erated  by  our  modified  HOT  model  (for  three  different 
(K,  (3)  pairs;  each  graph  contains  10,000  nodes).  Plot 
(a)  depicts  the  frequency  plot  of  the  number  of  locations 
per  node,  and  the  node  degree  frequency  plot  is  shown 
in  plot  (b).  In  ah  three  cases,  the  node  with  the  high¬ 
est  degree  (i.e.,  the  highest  ranked  node  in  Fig.  3(b))  has 
acquired  connections  to  a  majority  of  all  the  other  nodes 
(98.7%,  95.8%,  70.7%  of  ah  nodes,  resp.),  a  clear  indi¬ 
cation  that  the  resulting  graphs  exhibit  a  pronounced  star- 
shaped  structure.  For  comparison,  we  also  plot  in  Fig.  3(b) 
the  node  degree  distribution  produced  by  the  generic  HOT 
model  (i.e.,  a  single  location  per  node)  and  observe  that  in 
this  case,  the  node  with  the  highest  degree  is  connected  to 
only  14.5%  of  all  nodes.  This  result  confirms  our  hypothe¬ 
sis  and  illustrates  that  the  optimization  trade-off  that  is  the 
basis  of  the  generic  HOT  model  can  be  seriously  defeated 
in  a  more  realistic  AS  networking  setting. 


(a)  Connectivity  concentration  ratio 

Fig.  4.  Parameter  (K,{ 3)  exploration  in  the 

A  more  careful  exploration  of  the  parameter  space  (i.e., 

( K ,  /3)-values)  associated  with  the  modified  HOT  model 
further  confirms  our  conclusion.  We  define  the  connec¬ 
tivity  concentration  ratio  (CCR)  as  the  ratio  of  the  largest 
node  degree  over  the  total  number  of  links  in  the  graph. 
Using  CCR  as  a  metric  for  assessing  the  degree  of  “star¬ 
shapedness”  of  a  graph  (e.g.,  CCR=1  for  a  perfectly  star- 
shaped  topology),  we  examined  a  set  of  graphs  generated 
with  a  broad  range  of  (K,  (3) -values.  Fig.  4(a)  plots  the 
CCR  value  of  the  graphs  generated  by  the  modified  HOT 
model  with  0.0001  <  K  <0.1  and  0  <  j3  <  2.0.  Fig.  4(b) 
shows  the  largest  number  of  locations  per  node  in  the  cor¬ 
responding  graphs.  From  Fig.  4(a),  one  can  see  that  irre¬ 
spective  of  the  value  of  (3,  CCR  converges  quickly  to  1  as 
K  increases.  In  fact,  graphs  with  CCR-values  less  than  0.5 
or  so  are  only  possible  for  small  /v -values,  which,  accord¬ 
ing  to  Fig.  4(b),  give  rise  to  graphs  where  the  largest  num¬ 
ber  of  node  locations  is  only  about  10  or  less.  Note  that 
such  limited  variability  in  the  numbers  of  node  locations 
(small  K  values)  renders  the  modified  HOT  model  in¬ 
creasingly  indistinguishable  from  the  generic  HOT  model, 
where  each  node  has  only  a  single  location. 


IV.  AS  Geography  as  a  Causal  Force 

Motivated  by  networking  reality  (e.g.,  the  need  to  al¬ 
low  for  multi-PoP  ASs  with  realistic  geographies)  and  the 
shortcomings  of  the  HOT  model  of  Fabrikant  el  al.  dis¬ 
cussed  in  Section  III,  we  describe  in  the  following  the  first 
step  of  our  construction  of  a  new  HOT  model  for  Inter¬ 
net  growth  at  the  ASpc  level.  To  this  end,  we  distill  the 
role  that  AS  geography  plays  in  shaping  AS  connectiv¬ 
ity,  formulate  a  class  of  single-objective  optimizations  by 
which  newly  arriving  customer  ASs  select  their  upstream 
provider,  and  validate  the  findings  against  relevant  AS- 
specific  data. 


(b)  Largest  number  of  locations  per  node 


mo  difi  ed  HOT  model  (a=5  and  n=  10,000) 

A.  The  Univariate  HOT  Model 

To  account  for  the  multi-PoP  structure  of  real-world 
ASs,  we  rely  on  the  modified  HOT  model  described  in  Sec¬ 
tion  III-B,  but  redefine  the  local  optimization  problem  by 
which  an  individual  AS  connects  to  the  existing  AS  graph. 
In  particular,  viewing  nodes  as  customer  or  provider  ASs 
and  links  as  provider-customer  peering  relationships  and 
working  as  before  in  the  context  of  a  tree  topology,  we 
consider  the  single -objective  optimization  criterion  con¬ 
sisting  of  simply  minimizing  the  “last  mile”  connection 
cost.  Under  this  growth  model,  each  newly  arriving  node 
i  is  originally  identified  with  a  single  PoP  location;  upon 
arrival,  it  always  connects  to  the  existing  node  j  that  con¬ 
tains  the  PoP  location  in  locJist(j)  that  minimizes  the  Eu¬ 
clidean  distance  to  node  i.9  As  in  the  case  of  the  modified 
HOT  model,  each  existing  node  u  then  gets  a  chance  to 
enlarge  its  internal  PoP  structure  by  adding  a  new,  ran¬ 
domly  placed  PoP,  increasing  thereby  the  geographic  reach 
or  diversity  of  u.  For  each  node  u,  the  probability  to  in¬ 
crease  by  one  the  number  of  its  PoPs  in  locJist(u )  is  again 
given  by  pioc(u).  We  call  this  HOT  model  with  its  sin¬ 
gle  locality-based  connectivity  objective  the  univariate 
model  for  Internet  growth  at  the  AS  pc  level.10  It  is  ide¬ 
ally  suited  to  test  the  hypothesis  alluded  to  in  Section  III-B 
that  the  power-law  type  node  degree  distributions  of  in¬ 
ferred  AS  pc  or  AS  graphs  may  simply  be  due  to  the  pres¬ 
ence  of  power-law  type  distributions  that  capture  in  a  par¬ 
simonious  manner  the  high  variability  in  the  geographic 
extent  of  AS  infrastructures.  To  recall,  in  Section  III-B  we 
argued  that  due  to  the  proximity  to  the  topological  core 
of  the  AS  graph  and  the  geographic  diversity  of  their  PoP 
infrastructure,  large  ASs  arc  more  likely  to  acquire  new 
ASs,  which  in  turn  enables  them  to  build  up  their  PoP  in- 

9Being  an  existing  node,  node  j  will  in  general  already  possess  an 
extensive  internal  PoP  structure. 

10Univariate  refers  to  the  fact  that  geographic  proximity  is  the  only 
objective  that  is  being  optimized  by  a  newly  arriving  AS. 
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Number  of  Locations  per  Node 

(a)  Number  of  locations  per  node  (/?=0.1) 


Node  Degree 

(b)  Node  degree  (/3=0.1) 


Number  of  Locations  per  Node 

(c)  Number  of  locations  per  node  (/?=  1 .0) 


Node  Degree 

(d)  Node  degree  (J3=  1 .0) 


Fig.  5.  The  univariate  HOT  model  (n—10,000) 


frastructure  more  aggressively  than  small  ASs.  Together, 
these  arguments  make  it  plausible  that  high  variability  in 
the  number  of  PoPs  per  AS  may  well  cause  AS  degrees 
themselves  to  exhibit  high  variability. 

To  test  this  hypothesis  for  a  sensible  range  of  the  model 
parameters  K  and  ft  associated  with  our  univariate  HOT 
model,  we  generate  two  sets  of  graphs,  where  each  graph 
consists  of  10,000  nodes.  One  set  is  produced  by  setting 
ft  =  0.1  and  varying  K  between  0.0001  and  0.1,  the  other 
set  is  constructed  with  (3  =  1.0  and  the  same  K  range. 
The  top  row  of  Fig.  5  shows  the  distributions  of  the  num¬ 
ber  of  locations  per  node  in  plot  (a)  and  the  distributions 
of  node  degrees  in  plot  (b)  for  the  first  set  of  graphs  (i.e., 
/ 3  =  0.1).  The  bottom  row  of  the  figure  shows  the  cor¬ 
responding  information  in  plots  (c)  and  (d)  for  the  second 
set  of  graphs  (i.e.,  (3  =  1.0).  Clearly,  none  of  the  distri¬ 
butions  associated  with  the  first  set  of  graphs  exhibit  high 
variability  or  power-law  type  tails.  In  fact,  all  the  distribu¬ 
tions  show  extremely  limited  variability,  fully  consistent 
with  exponential-type  distributions.  In  particular,  we  ob¬ 
serve  that  for  graphs  with  (3  =  0.1,  the  exponential-type 
distributions  for  the  number  of  locations  per  node  are  in¬ 
capable  of  producing  highly  variable  node  degree  distribu¬ 
tion.  However,  graphs  with  ft  =  1.0  (bottom  row)  give 
rise  to  a  more  interesting  behavior.  As  the  value  of  K 
increases  from  0.0001  to  0.1,  not  only  does  the  number 


of  locations  per  node  become  more  highly  variable,  but 
the  corresponding  node  degree  distributions  also  change 
from  exhibiting  very  limited  variability  (exponential-type) 
to  showing  high  variability  of  the  power-law  type.  In  par¬ 
ticular,  for  the  larger  /\ -values  (e.g.,  0.01  <  K  <  0.1), 
there  is  a  striking  resemblance  between  the  distributions 
of  the  number  of  locations  per  node  and  the  node  degrees. 
From  this  experiment,  we  conclude  that  our  hypothesis  has 
merit;  that  is,  the  geographic  extent  of  existing  nodes  (as 
measured  by  the  number  of  locations)  can  indeed  be  a  con¬ 
trolling  force  in  determining  the  characteristics  of  node  de¬ 
gree  distributions.  This  finding  motivates  a  careful  investi¬ 
gation  into  the  geographic  properties  of  existing  ASs,  with 
the  ultimate  goal  of  validating  the  high  variability  of  AS 
size  (measured  in  terms  of  the  number  of  PoPs  per  AS) 
through  elementary,  geography-related  measurements. 

B.  On  Validating  the  Univariate  HOT  Model 

The  proposed  univariate  HOT  model  for  Internet  growth 
at  the  AS  pc  level  suggests  that  the  high  variability  of  the 
AS  sizes  as  measured  by  the  number  of  PoPs  is  an  expla¬ 
nation  for  the  striking  power-law  type  AS  degree  distri¬ 
butions  of  inferred  ASpc  graphs  and,  in  turn,  of  the  AS 
graphs  themselves.  To  validate  this  aspect  of  the  model 
and  “close  the  loop”  in  the  sense  of  [32],  we  provide  be¬ 
low  empirical  evidence  for  the  ubiquity  of  high  variability 
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in  the  geographic  extent  of  existing  ASs.  In  particular,  we 
present  and  discuss  in  the  following  the  types  of  measure¬ 
ments  and  analysis  methodologies  that  enable  us  to  charac¬ 
terize  various  aspects  of  existing  AS  geographies,  includ¬ 
ing  the  distribution  of  the  number  of  PoPs  per  AS. 

To  infer  AS  size  in  terms  of  the  number  of  PoPs,  we 
first  need  to  infer  the  geographic  locations  of  the  PoPs  of 
the  different  ASs.  To  this  end,  we  first  collected  a  set  of 
address  prefixes  from  the  Oregon  route-views  BGP  data. 
Since  BGP-advertised  address  prefixes  can  contain  over¬ 
lapping  address  space,  to  be  able  to  associate  address  space 
with  geographic  location,  we  converted  any  overlapping 
address  prefixes  to  a  set  of  disjoint  address  blocks  by  re¬ 
cursively  splitting  in  half  any  prefix  that  contains  a  sub¬ 
prefix.  For  example,  given  the  overlapping  address  space 
128.182.0.0/16  and  128.182.64.0/18,  the  recursive  split¬ 
ting  procedure  generates  the  three  disjoint  address  blocks 
128.182.128.0/17,  128.182.64.0/18,  and  128.182.0.0/18. 
With  110,281  BGP-advertised  address  prefixes  from  the 
Oregon  BGP  data,  after  applying  the  recursive  splitting, 
we  were  left  with  a  total  of  197,841  disjoint  CIDR  blocks. 

Next,  to  associate  a  geographic  location  with  each  of 
these  disjoint  CIDR  blocks,  we  randomly  picked  an  IP 
address  from  each  of  the  197,841  blocks  and  queried  the 
NetGeo  database,  a  public  repository  of  geographic  infor¬ 
mation  associated  with  address  prefixes  [10],  for  its  geo¬ 
graphic  location.  The  NetGeo  server  responded  with  geo¬ 
graphic  location  records  for  about  97%  of  all  the  queries 
we  made.  Relying  on  all  these  NetGeo  records,  we  then 
map  each  of  the  geographically  known  IP  address/CIDR 
block  to  its  corresponding  AS,  thereby  producing  the  geo¬ 
graphic  mapping  information  for  the  individual  ASs. 

While  measuring  the  geographic  extent  of  existing  ASs 
by  the  number  of  their  distinct  NetGeo-inferred  loca¬ 
tions,  we  found  that  the  geographic  granularity  of  the  Net¬ 
Geo  data  is  inconsistent,  both  with  respect  to  ( longitude , 
latitude )  coordinates  and  {city! state! country)  informa¬ 
tion.  1 1  To  ensure  that  our  inference  methods  are  not  biased 
or  invalidated  by  these  inconsistencies,  we  use  a  more  flex¬ 
ible  and  practical  definition  of  geographic  granularity  by 
relying  on  the  following  heuristic.  For  each  AS,  using  the 
corresponding  inferred  geographic  mapping  information, 
we  first  obtain  the  ( longitude ,  latitude)  coordinates  of  all 
of  its  distinct  locations.  Then  we  merge  locations  whose 
pairwise  geographic  distances  are  below  a  given  threshold 

nFor  example,  since  NetGeo  infers  US  locations  from  zip  code  or 
phone  area  code  information  (typically  of  fi  ner  granularity  than  city 
names),  San  Francisco  has  40  distinct  ( longitude ,  latitude )  locations 
in  the  NetGeo  data  set,  but  they  differ  only  slightly  from  one  another. 
On  the  other  hand,  for  most  non-US  cities,  NetGeo  assigned  only  a 
single  geographic  location. 


TABLE  III 

Rocketfuel  vs.  NetGeo 


AS 

Name 

Number  of  PoPs  | 

Rocketfuel 

NetGeo  (r  =  100  mi) 

1221 

Telstra 

61 

16 

1239 

Sprintlink 

43 

302 

1755 

Ebone 

25 

13 

2914 

Verio 

121 

51 

3257 

Tiscali 

50 

3 

3356 

Level3 

52 

42 

3967 

Exodus 

23 

36 

4755 

VSNL 

10 

35 

6461 

Abovenet 

21 

23 

7018 

AT&T 

108 

245 

distance  r  into  one  cluster  or  location- group  and  use  the 
arithmetic  mean  of  the  ( longitude ,  latitude)  coordinates 
of  the  individual  locations  that  got  merged  as  the  cluster’s 
new  coordinates.  Thus,  by  clustering  closely  linked  ge¬ 
ographic  locations,  our  method  is  expected  to  recover,  at 
least  to  a  first  order,  the  actual  physical  facilities  of  existing 
ASs.  We  will  use  in  the  following  the  number  of  inferred 
location-groups  per  AS  as  our  estimate  for  AS  size. 

A  number  of  recent  studies  have  been  concerned  with 
inferring  the  geographic  locations  of  existing  network- 
related  entities  such  as  Internet  hosts,  IP  routers,  PoPs, 
ISPs,  or  ASs  [26],  [22],  [29],  [19],  [10],  Of  particular 
relevance  is  [29],  where  the  authors,  as  paid  of  obtain¬ 
ing  some  of  the  most  complete  currently  available  pub¬ 
lic  router-level  maps  for  10  existing  ASs,  also  inferred  the 
number  of  PoPs  for  those  same  10  ASs.  To  compare,  Ta¬ 
ble  III  shows  the  inferred  number  of  PoPs  reported  in  [29] 
(left  column)  and  obtained  using  our  NetGeo-based  ap¬ 
proach  (right  column).  Clearly,  the  differences  are  sig¬ 
nificant  and  beg  an  explanation.  Consider,  for  example, 
an  extreme  case  like  AS  1239  (SprintLink),  where  [29] 
reports  43  PoPs  while  our  method  yields  302  (with  r  = 
100  mi).  To  understand  why  such  differences  arc  to  be  ex¬ 
pected,  first  note  that  the  geographic  granularity  of  the  two 
methods  is  different.  While  [29]  uses  city  names,  the  geo¬ 
graphic  granularity  of  our  method  can  be  finer  or  coarser, 
depending  on  the  choice  of  the  threshold  distance  parame¬ 
ter  t.  Second,  a  design  feature  of  SprintLink  is  that  small 
city  customers  tend  to  be  back-hauled  to  far  away  PoPs 
(typically  located  in  the  major  cities)  through  geographi¬ 
cally  close -by  layer-2  switches  which  remain  invisible  to 
the  approach  pursued  in  [29],  but  which  arc  likely  to  in¬ 
flate  the  number  of  inferred  PoPs  when  using  our  method 
(i.e.,  our  method  may  wrongly  infer  the  locations  of  such 
layer-2  switches  as  actual  PoP  locations).12 

12It  could  be  argued  that  for  the  purposes  of  this  paper,  layer-2 
switches  in  a  SprintLink-like  network  design  should  be  viewed  and 
counted  as  PoPs. 
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Fig.  6.  AS  size  distribution  (size  =  number  of  PoPs). 


One  way  to  mitigate  the  inherently  difficult  problem 
of  inferring  geographic  locality  information  for  network 
elements  (e.g.,  number  of  PoPs  per  AS)  from  BGP-  or 
traceroute-derived  measurements  is  to  demonstrate  that  the 
obtained  information  is  broadly  robust.  To  illustrate,  con¬ 
sider  the  inferred  AS  size  obtained  using  our  NetGeo- 
based  technique.  Plot  (a)  in  Fig.  6  depicts  the  distributions 
of  the  inferred  AS  sizes  for  different  values  of  the  thresh¬ 
old  distance  parameter  r  (i.e.,  10,  20,  50,  and  100  mi)  and 
shows  that  AS  size  distribution  as  measured  by  the  num¬ 
ber  of  PoPs  per  AS  is  largely  insensitive  to  the  choice  of 
r.  Plot  (b)  in  Fig.  6  demonstrates  an  even  stronger  ro¬ 
bustness  property  of  the  AS  size  distribution.  To  explicitly 
account  for  possible  PoP  count  inflation  (as  in  the  case  of 
SprintLink)  or  deflation  (e.g.,  too  coarse  of  a  geographic 
granularity),  we  assume  that  Rerr%  of  all  the  ASs  have 
their  sizes  wrongly  inferred  by  our  NetGeo  technique.  We 
randomly  mark  half  of  them  as  having  their  sizes  inflated, 
with  the  other  half  having  deflated  sizes.  For  each  AS  with 
an  inflated  size,  we  “adjust”  its  size  by  multiplying  it  by 
1/e,  where  e  is  randomly  chosen  from  [1,  emax]  and  where 
emax  is  a  parameter  indicating  some  maximum  margin  of 
error.  On  the  other  hand,  for  each  deflated  AS,  we  “mod¬ 
ify”  its  size  by  multiplying  it  by  a  factor  of  e,  where  e  is 
again  picked  at  random  from  [1 ,  emax] .  For  example,  if  the 
inferred  size  of  AS  X  is  100  and  emax  =  4,  the  adjusted 
size  of  AS  X  can  vary  between  25  and  100  if  X  is  con¬ 
sidered  inflated,  and  between  100  and  400  if  it  is  marked 
deflated.  Assuming  that  the  inferred  AS  size  can  deviate 
from  the  (unknown)  “true”  size  by  as  much  as  a  factor  of 
4  (i.e.,  emax  =  4),  plot  (b)  in  Fig.  6  testifies  to  the  ro¬ 
bustness  of  the  power-law  type  behavior  of  the  AS  size 
distribution  with  respect  to  a  wide  range  of  misspecifica- 
tions  of  AS  sizes  (i.e.,  Rerr  =  20%,  40%,  60%,  80%,  and 
even  for  Rerr  =  100%).  Note  that  such  robustness  prop¬ 
erties  indicate  an  enormous  resilience  to  ambiguity  of  the 
unknown  underlying  distribution  (e.g.,  ambiguity  with  re¬ 


spect  to  precisely  how  AS  size  is  defined  or  measured)  and 
essentially  imply  that  the  latter  must  be  of  the  power-law 
type  [9].  Based  on  these  empirical  findings,  we  conclude 
that  the  power-law  type  distribution  for  AS  sizes  that  has 
been  identified  by  our  univariate  HOT  model  for  Internet 
growth  at  the  ASpc  level  as  an  explanation  for  the  power- 
law  type  node  degree  distributions  of  inferred  ASpc  and 
AS  graphs  is  fully  consistent  with  data  derived  from  geo¬ 
graphic  mapping  information  for  the  individual  ASs. 

C.  Some  comments  on  using  the  NetGeo  database 

[22]  reports  on  a  study  of  the  geographic  population  of 
Internet  routers  that  were  discovered  by  traceroute  prob¬ 
ing.  That  work  relies  on  proprietary  geographic  location 
databases  to  infer  the  coordinates  of  all  router  interface 
addresses.  The  work  described  here  differs  from  [22]  in 
a  number  of  ways.  For  one,  as  discussed  in  Section  4.2 
above,  instead  of  considering  an  inevitably  incomplete  set 
of  router  addresses  obtained  from  traceroute  probing,  we 
examine  the  locations  associates  with  all  existing  address 
prefixes.  Second,  we  rely  on  a  public  repository  of  geo¬ 
graphic  information  for  IP  address  prefixes,  i.e.,  the  Net¬ 
Geo  database  [10]. 

A  typical  NetGeo  record  is  shown  below  and  contains 
(city/ state! country)  information  and  the  ( latitude/longitude ) 
coordinates  for  the  (target)  IP  address  in  question. 

LONG:  -77.11 

LAT:  38.87 

STATE:  VIRGINIA 

NAME:  CHY-WAN-65-121-98A 

STATUS :  OK 

COUNTRY:  US 

LAT_LONG_GRAN :  City 

TARGET:  65.121.98.28 

NUMBER:  65.121.98.0  -  65.121.98.255 

CITY:  ARLINGTON 

NIC:  ARIN 

LAST_UPDATED :  14-NOV-2001 
LOOKUP_TYPE:  Block  Allocation 
DOMAIN_GUESS :  qwestip.net 


12 


The  geographic  location  records  of  the  NetGeo  database 
originate  from  the  ARIN/APNIC/RIPE  whois  servers.  The 
NetGeo  database  server  extracts  geographic  information 
such  as  city,  state,  country  names  from  the  text  of  whois 
records.  It  also  leverages  US  zip  codes,  phone  area  codes, 
or  email  addresses  with  2-letter  top-level  domains  (TLDs) 
country  code  where  possible.  For  large,  geographically- 
dispersed  transit  providers  for  which  the  whois-based  tech¬ 
nique  is  not  sufficient  for  location  mapping,  the  NetGeo 
obtains  geographical  hints  from  DNS  lookup  as  well. 

Given  that  NetGeo’s  inference  engine  exploits  various 
geographic  cues,  we  performed  a  careful  analysis  to  cal¬ 
ibrate  its  mapping  accuracy.  The  following  are  the  main 
conclusions  of  our  assessment  of  the  reliability  of  the  Net¬ 
Geo  data. 

•  Comparison  with  Geotrack  ( US  sites  only):  Using  1,000 
US  university  sites  with  known  geographic  locations,  corn- 
paling  their  NetGeo-inferred  locations  with  the  corre¬ 
sponding  locations  inferred  by  the  Geotrack  method  (see 
for  example  [26]),  we  found  that  the  NetGeo-based  loca¬ 
tions  arc  slightly  more  accurate  than  those  produced  by 
Geotrack  (while  the  median  difference  between  the  former 
and  the  actual  locations  was  0,  it  was  about  80  miles  for 
the  Geotrack-related  differences). 

•  Comparison  with  Geotrack  ( random  sites):  Using  660 
randomly  chosen  IP  addresses  whose  actual  geographic 
locations  are  typically  not  known,  we  observed  that  the 
NetGeo-  and  Geotrack-inferred  locations  vary  widely, 
with  as  many  as  40%  of  the  IP  addresses  having  NetGeo- 
and  Geotrack-inferred  locations  that  arc  more  than  1,000 
miles  apart  from  one  another.  It  turns  out  that  the  rea¬ 
son  for  this  extreme  discrepancy  between  the  two  infer¬ 
ence  methods  is  that  Geotrack  has  a  tendency  to  associate 
the  geographic  location  of  networks  that  reside  outside  the 
US  with  the  location  of  US  cities  where  the  the  networks’ 
international  circuits  originate  or  terminate. 

•  Comparison  with  Geotrack  (random  US  sites):  When 
considering  only  those  random  IP  addresses  (out  of  the 
above  660)  which  -  according  to  NetGeo  -  arc  located 
within  the  US,  we  found  that  the  NetGeo-  and  Geotrack- 
inferred  locations  arc  now  more  comparable,  with  a  me¬ 
dian  difference  of  about  100  miles. 

As  described  in  Section  4.2,  because  of  inconsistencies 
in  the  geographic  granularity  of  the  NetGeo  data,  we  con¬ 
sider  a  flexible  and  very  practical  definition  of  geographic 
granularity  that  essentially  ignores  location  information 
that  is  too  fine-grained  (i.e.,  below  a  threshold  distance  of 
r  miles).  As  a  by-product,  this  definition  can  also  mitigate 
the  problem  whereby  portions  of  address  space  belonging 
to  an  ISP  X  may  be  delegated  to  other  business  customers 
that  obtain  their  network  access  from  X  without  running 


Location  Density 

Fig.  7.  Location  density  distribution  (density  =  number  of  ASs). 

BGP  themselves  (e.g.,  managed  hosting  service  clients  or 
small  dial-up/broadband  access  sites).  In  this  case,  the 
NetGeo  data  set  may  associate  the  locations  of  such  re¬ 
assigned  address  blocks  with  those  of  these  business  cus¬ 
tomers,  causing  the  location  of  ISP  X  to  be  wrongly  ex¬ 
tended  to  the  locations  of  those  customer  sites.  We  predict 
that  such  business  clients  would  typically  be  located  in  ge¬ 
ographic  proximity  to  their  provider  ISP’s  actual  locations 
due  to  the  economic  incentive  of  laying  shorter  lines  or  the 
ease  of  administrative  contact.  In  any  case,  the  heuristic  is 
likely  to  track  the  locations  of  the  physical  facilities  of  a 
given  AS  with  reasonable  accuracy. 

We  conclude  this  section  with  another  example  of  how 
the  NetGeo  data  can  be  used  to  study  interesting  fea¬ 
tures  associated  with  AS  geography.  Recall  that  the  de¬ 
velopment  of  our  locality-basd  HOT  model  for  Internet 
growth  at  the  AS pc  level  was  partly  motivated  by  the  ar¬ 
gument  that  large  multi-PoP  ASs  arc  more  likely  to  ac¬ 
quire  new  customers  than  single-PoP  or  small  multi-PoP 
ASs.  Knowing  the  geographic  concentration  of  existing 
ASs  (i.e.,  the  number  of  different  ASs  per  geographic  lo¬ 
cation)  would  shed  light  on  the  validity  of  the  above  ar¬ 
gument.  Thus,  to  obtain  the  number  of  ASs  per  geo¬ 
graphic  location,  we  first  collect  from  our  NetGeo  data 
set  the  ( longitude ,  latitude)  coordinates  of  all  the  dis¬ 
tinct  geographic  locations  that  arc  associated  with  at  least 
any  one  AS.  As  before,  depending  on  the  threshold  dis¬ 
tance  r,  we  group  all  the  pairwise  close  geographic  lo¬ 
cations  into  one  cluster  or  location-group  and  then  asso¬ 
ciate  individual  ASs  with  the  different  location-group  as 
follows:  AS  i  is  associated  with  a  given  location-group 
if  and  only  if  AS  i  has  at  least  one  geographic  location 
that  is  contained  in  the  location  group.  When  equating  the 
number  of  ASs  per  location  with  the  inferred  number  of 
ASs  per  location-group.  Fig.  7  shows  the  resulting  distri¬ 
butions  for  four  different  values  of  the  threshold  distance 
r  (i.e.,  r  =10,  20,  50,  and  100  miles).  Additional  details 
about  the  top- 10  locations  in  terms  of  the  number  of  ASs 
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TABLE  IV 

Top- 10  locations  (r  =  10  miles) 


Location  Rank 

Location  Name 

Number  of  ASs 

i 

New  York  City,  NY,  US 

508 

2 

Herndon,  VA,  US 

411 

3 

Amsterdam,  NL 

366 

4 

Fairfax,  VA.  US 

316 

5 

Englewood,  CO.  US 

313 

6 

Seoul,  KR 

310 

7 

Palo  Alto,  CA,  US 

269 

8 

Atlanta,  GA,  US 

245 

9 

Los  Angeles,  CA 

225 

10 

London,  UK 

211 

present  can  be  found  in  Table  IV.  Not  surprisingly,  the 
top- 10  locations  correspond  mainly  to  large  cities  or  ma¬ 
jor  urban  areas,  with  high  population  densities  and  signif¬ 
icant  economic  activities.13  From  Fig.  7  we  can  conclude 
that  the  geographic  concentration  of  existing  ASs  exhibits 
high  variability  (ranging  over  3  orders  of  magnitude)  and 
is  largely  insensitive  to  the  details  of  defining  geographic 
granularity  (i.e.,  specific  choice  of  r). 

V.  AS  Business  Model  as  a  Causal  Force 

Building  upon  our  univariate  HOT  model,  we  describe 
in  the  section  the  second  step  of  or  construction  of  a  new 
HOT  model  for  Internet  growth  at  the  AS  pc  level.  In  par¬ 
ticular,  we  explore  here  in  detail  the  role  that  the  decision 
making  processes  by  which  the  different  customer  ASs  en¬ 
ter  into  business  relationships  with  their  provider  ASs  play 
in  shaping  Internet  AS  connectivity. 

A.  Inter-AS  Peering  in  the  Commercial  Internet 

By  our  univariate  HOT  model,  the  variability  in  AS  size 
roughly  determines  the  variability  in  AS  node  degrees  (see 
Fig.  5).  However,  while  inferred  AS  graphs  of  the  actual 
Internet  (e.g..  Fig.  1)  typically  show  3  to  4  orders  of  mag¬ 
nitude  variability  in  node  degrees,  the  variability  of  the 
inferred  number  of  PoPs  per  AS  (i.e.,  AS  sizes)  tends  to 
range  over  only  2  to  3  orders  of  magnitude  (see  Fig.  6). 
While  this  suggests  that  managing  ASs  with  too  many 
PoPs  is  economically  or  technically  not  a  viable  business, 
it  is  also  a  clear-  indication  that  our  univariate  HOT  model 
has  only  identified  a  necessary  but  by  no  means  sufficient 
condition  for  explaining  the  high  variability  of  inferred 
AS  node  degrees.  The  model’s  claim  is  fully  consistent 
with  elementary,  AS  geographic  measurements,  but  it  does 
not  rule  out  the  existence  of  factors  other  than  geographic 
proximity  at  work  in  shaping  AS  connectivity. 

One  such  factor  is  how  a  new  customer  AS  X  es¬ 
tablishes  Internet  connectivity  when  there  are  numerous 

13As  we  vary  r  from  10  miles  to  100  miles,  we  notice  a  slight  re¬ 
ordering  among  the  top-10  ranking,  but  the  ASs  remain  the  same. 


provider  ASs  within  the  same  (or  comparable)  geographic 
proximity  of  X,  all  vying  to  become  AS  2f’s  provider. 14 
In  such  situations,  it  is  reasonable  to  assume  that  AS  X 
will  evaluate  the  competing  providers  by  several  criteria 
other  than  geographic  proximity  (including  for  example 
the  availability,  reliability,  and  performance  of  the  net¬ 
works,  available  pricing  plans,  existing  customer  support, 
number  of  value-added  services,  geographic  reach  and 
projected  network  build-out,  prior  acquaintance  between 
the  parties  involved,  etc.  [23])  and  then  make  a  more  or  less 
rational  decision  to  choose  a  provider  that  is  “best”  or  “op¬ 
timal”  with  respect  to  some,  possibly  AS-dependent,  util¬ 
ity  measure.  To  formalize  this  admittedly  over-simplified 
process  by  which  new  customer  ASs  select  their  upstream 
provider,  we  view  the  above  criteria  as  being  part  of  an  ab¬ 
stract  object  called  AS  X ’s  business  model.  Aside  from 
the  above  mentioned  criteria,  the  business  model  of  an  AS 
can  include  any  other  factors  that  may  play  a  role  in  how 
AS  connectivity  is  established.  In  Section  V-B  below,  we 
present  a  concrete  mathematical  formulation  of  such  toy 
business  models,  flexible  enough  to  account  for  a  range  of 
different  business  objectives  among  the  various  ASs. 

B.  The  Bivariate  HOT  Model 

In  incorporating  an  AS’s  business  model  into  our  pro¬ 
posed  HOT  model,  we  assume  that  a  newly  arriving 
AS  first  identifies  within  its  geographic  neighborhood  ah 
provider  ASs  offering  service.  The  candidate  providers  are 
then  evaluated  based  on  a  set  of  criteria  in  the  AS ’s  busi¬ 
ness  model.  Finally,  the  “best”  upstream  provider  for  the 
new  AS  is  selected  as  the  result  of  the  AS’s  locally  optimal 
business  decision.  More  formally,  we  start  with  the  uni¬ 
variate  model  defined  in  Section  IV-A,  where  each  node  i 
has  a  list  locJist(i )  that  provides  geographic  information 
about  its  internal  PoP  structure  (the  number  of  PoPs  and 
their  geographic  coordinates).  To  define  node  V s  business 
model,  we  augment  node  i  with  an  iV-dimensional  vec¬ 
tor  x i  =  (ay;i,  Xit2,  ay,jv)  and  a  non-empty  set  Si  C 
{1,2,..., TV}.  We  call  ay  node  T s  score  vector.  An  instan¬ 
tiation  of  ay  is  obtained  by  assigning  each  of  its  compo¬ 
nents  ay ,n  a  uniform  random  number  taken  from  [0,1].  .S) 
is  node  T s  selection  set  and  and  is  randomly  chosen  from 
2{t, 2,...,N}  \  The  pajr  (a^g^)  forms  node  V s  busi¬ 

ness  model.  Intuitively,  the  score  vector  indices  represent 
the  possible  criteria  that  enter  into  the  decision  making 
processes  by  which  competing  ASs  are  evaluated  against. 
The  individual  vector-components  ayj  quantify  how  AS  i 

14The  importance  of  this  factor  in  establishing  provider-customer 
peering  relationships  among  ASs  is  emphasized  by  the  observed  highly 
variable  AS  density  per  geographic  location  as  discussed  at  the  end  of 
Section  I  V-B. 
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Fig.  8.  The  bivariate  HOT  model  (n=  1 0,000,  I\=  0.05,  (3=  1.0,  neighborhood  radius=0.1,  N=3) 


is  measured  up  with  respect  to  criterion  3,  1  <  3  <  N. 
The  selection  set  Si  defines  the  subset  of  criteria  that  AS 
i  deems  relevant  when  choosing  its  upstream  provider. 
Hence  in  making  its  choice  for  a  provider,  a  new  AS  i 
matches  up  its  selection  set  against  the  score  vector  of  each 
candidate  provider.  The  selection  set  is  generally  different 
from  one  AS  to  another. 

To  determine  the  comparative  desirability  of  provider 
ASs,  we  define  node  dominance  on  a  set  P  as  follows. 
Given  two  nodes  i  and  j  and  a  non-empty  set  P  C 
{1, 2, ...,  N},  node  i  is  said  to  dominate  node  j  on  set 
P  if  Xi,n  >  Xj)n,  for  all  n  €  P,  and  >  xy,m,  for 
some  rn  G  P.  To  illustrate,  consider  the  following  sim¬ 
ple  example  whereby  three  competing  provider  ASs,  AS 
X,  Y,  and  Z,  arc  evaluated  by  a  newly  arriving  customer 
AS  i  in  terms  of  two  (N  =  2)  criteria:  network  reliabil¬ 
ity  and  unit  bandwidth  cost.  AS  X  reportedly  has  99% 
network  reliability,  AS  Y  98%,  and  AS  Z  97%.  They 
charge  $  100/mon.,  $  150/mon.  and  $50/nron.  per  unit  band¬ 
width,  respectively.  If  the  new  customer  AS  i  considers 
unit  bandwidth  cost  as  the  only  relevant  criterion  in  choos¬ 
ing  a  provider  (i.e.,  Si  =  {unit  bandwidth  cost}),  then  it 
will  choose  AS  Z.  However,  a  new  customer  AS  j  that 
deems  both  network  reliability  and  unit  bandwidth  cost 
important  when  selecting  its  upstream  provider  (i.e.,  Sj  = 
{network  reliability,  unit  bandwidth  cost})  will  not  select 
AS  Y,  since  AS  Y  is  dominated  by  AS  X.  It  will  choose 
either  AS  X  or  Z,  for  neither  is  dominated  by  the  other 
(ties  can  be  broken  as  j  pleases).  Note  that  in  this  process, 
the  score  vector  of  a  newly  arriving  customer  itself  is  not 
needed,  but  its  selection  set  is  crucial;  at  the  same  time, 
any  newly  arriving  customer  AS  needs  to  know  the  score 
vector  xu  and  geography  data  locdist{u )  of  each  existing 
provider  AS  u  in  the  graph. 

According  to  this  model,  the  graph  grows  as  follows. 
When  a  new  node  i  arrives,  it  is  assigned  the  trivial  list 
locJist(i)  (consisting  of  i’ s  coordinates  only)  and  an  in¬ 


stantiation  of  its  selection  set  Si.  Node  i  first  initializes 
its  candidate  set  candidate(i)  with  all  existing  nodes  that 
have  a  PoP  location  within  a  pre-defined  Euclidean  dis¬ 
tance  from  node  i,  the  so-called  neighborhood  of  i.  Next, 
for  each  pair  of  nodes  in  candidate{i) ,  if  one  of  the  nodes 
is  dominated  on  the  set  Si  by  the  other  node,  the  new  node 
i  marks  that  dominated  node.  Subsequently,  all  marked 
nodes  are  eliminated  from  candidate{i) .  The  new  node 
then  randomly  picks  one  of  the  nodes  left  in  candidate(i) 
as  the  target  node  and  establishes  a  connection  to  that  node. 
Finally,  node  i  is  given  an  instantiation  of  its  score  vector 
Xi,  and — as  in  the  multi-PoP  univariate  HOT  model — each 
existing  node  u  is  given  a  chance  to  increments  the  number 
of  its  PoP  locations  in  locdist(u)  by  1,  with  probability 
Ploc{u)\  with  probability  1  —  pioc{u),  locJist(u)  is  left  un¬ 
changed.  We  call  this  HOT  model  in  which  two  objec¬ 
tives  (i.e.,  geographic  proximity  and  economic  utility) 
are  optimized  simultaneously  the  bivariate  model  for  In¬ 
ternet  growth  at  the  ASpc  level.  This  model  differs  from 
the  previously  considered  univariate  model  in  that  a  new 
AS  i,  instead  of  finding  the  geographically  closest  AS,  now 
considers  all  the  ASs  with  PoP  locations  “close-by,”  i.e., 
located  within  its  neighborhood.  More  importantly,  when 
trying  to  narrow  down  its  choices  among  multiple  locally 
accessible  candidate  provider  ASs,  the  new  AS  i  avoids  se¬ 
lecting  a  provider  that  is  dominated  by  some  other  compet¬ 
ing  provider  in  terms  of  the  criteria  present  in  its  selection 
set  Si.  The  model  ensures  that  the  final  selection  of  up¬ 
stream  provider  by  each  new  AS  i  is  Pareto  optimal  [16] 
in  the  sense  that  no  other  “close-by”  provider  AS  domi¬ 
nates  the  chosen  provider  on  the  set  St.  This  HOT  formu¬ 
lation  is  a  truly  multi-objective  optimization,  combining 
geography-  and  economic-specific  objectives.  Moreover, 
it  defines  a  fully  distributed  and  decentralized  design  of 
Internet  connectivity  at  the  AS' pc  level,  typically  with  dif¬ 
ferent  optimization  problems  (as  a  result  of  the  different 
AS-specific  selection  sets)  solved  by  the  different  ASs. 
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Fig.  9.  The  bivariate  HOT  model  (n=  10,000,  A'=0.05,  /3=1.0)  -  (a)  the  effect  of  neighborhood  radius  (for  fixed  N= 3),  (b)  the 
effect  of  the  dimension  of  the  score  vector  (for  fi  xed  neighborhood  radius=0.1). 


To  demonstrate  the  potential  of  this  bivariate  HOT 
model.  Fig.  8  illustrates  some  of  the  features  of  the  re¬ 
sulting  graphs  (as  always,  each  graph  consist  of  10,000 
nodes).  In  particular,  defining  the  neighborhood  of  a  given 
node  to  be  a  disk  of  radius  0. 1  centered  at  a  node,  setting 
K  =  0.05,  /?=1.0,  and  N  =  3,  plot  (a)  of  Fig.  8  depicts 
the  distribution  of  the  number  of  locations  per  node,  and 
plot  (b)  gives  the  node  degree  distribution  of  the  generated 
graphs.  On  the  same  plots,  we  also  show  the  results  cor¬ 
responding  to  the  graph  generated  by  the  univariate  model 
(same  parameters:  K  =  0.05,  (3=  1.0,  10,000  nodes).  Fi¬ 
nally,  for  comparison,  plot  (b)  also  contains  the  node  de¬ 
gree  distribution  corresponding  to  one  of  the  single-homed 
provider-customer  subgraph  of  the  OREGON+  AS  graph 
(i.e.,  the  largest  tree  topology  rooted  at  AS701;  see  Sec¬ 
tion  II-C).  Fig.  8  provides  convincing  evidence  that  while 
the  univariate  and  bivariate  HOT  models  yield  compar  able 
AS  size  distributions,  the  latter  has  a  clear-  impact  on  the 
distribution  of  node  degrees.  In  fact,  it  can  increase  the 
variability  of  node  degrees  by  at  least  one  order  of  mag¬ 
nitude  beyond  what  the  univariate  model  is  capable  of, 
closely  matching  the  node  degree  distribution  of  inferred 
AS  (sub)graphs. 

Next  we  illustrate  that  both  objectives  in  the  formula¬ 
tion  of  our  bivariate  HOT  model  (i.e.,  geographic  proxim¬ 
ity  and  locally  Pareto-optimal  business  decision)  are  nec¬ 
essary  for  creating  graphs  with  Internet-like  features.  We 
first  examine  in  plot  (a)  of  Fig.  9  how  the  geography- 
related  factor,  i.e.,  the  neighborhood  of  a  node,  impacts 
the  node  degree  distribution  of  the  resulting  final  graph.  To 
this  end,  we  set  AT=0.05 ,  (3=  1 .0,  and  N=3,  but  vary  the  ra¬ 
dius  of  a  node’s  neighborhood  from  0.01,  0.1,  0.2,  0.5,  and 
some  large  number.  As  can  be  observed,  as  the  neighbor¬ 
hood  size  decreases  to  a  disk  of  radius  0.01,  the  resulting 
node  degree  distributions  exhibit  less  variability  and  be¬ 
come  comparable  to  those  generated  by  similarly  parame¬ 
terized  versions  of  our  univariate  models  (see  for  example, 


Fig.  8(a)).  On  the  other  hand,  too  large  of  a  neighbor¬ 
hood  size  (e.g.,  neighborhood  sizes  >  0.2)  results  in  degree 
distributions  that  are  qualitatively  different  from  those  de¬ 
rived  from  the  actual  Internet.  These  observations  suggest 
that  a  certain  degree  of  geographic  locality  is  needed  to 
model  Internet-like  ASpc  graphs  with  our  bivariate  HOT 
model. 

To  examine  how  the  business-related  criterion  affects 
node  degree  distributions,  we  parameterize  our  model  with 
K  =  0.05,  (3=  1.0,  and  a  neighborhood  radius  of  0.1,  but 
vary  the  parameter  N,  the  dimension  of  the  score  vector, 
from  1  to  9.  N  captures  the  complexity  of  the  business- 
related  aspect  of  provider  selection  and  is  therefore  a  nat¬ 
ural  parameter  to  consider  for  the  purposes  of  this  experi¬ 
ment.  Plot  (b)  of  Fig.  9  shows  the  node  degree  distributions 
of  a  10,000-node  graph,  for  the  different  values  of  N.  As 
the  value  of  N  increases,  the  variability  of  node  degrees 
tends  to  decrease.  This  observation  agrees  with  the  intu¬ 
ition  that  a  small  Ar-valuc  creates  more  opportunities  for 
some  nodes  to  be  favored  over  the  others  during  the  growth 
process  of  the  graph,  which  in  turn  increases  the  variabil¬ 
ity  of  node  degrees.  Small  N -values  reflect  the  regime 
whereby  relatively  simple  business  decisions  are  involved 
in  selecting  the  “best”  provider.  If  complex  business  deci¬ 
sions  are  allowed  (large  N- values),  the  cardinality  of  the 
Pareto-optimal  candidate  set  is  likely  to  be  large,  caus¬ 
ing  individual  node  degrees  to  be  more  evenly  distributed 
among  existing  nodes,  thereby  reducing  the  overall  node 
degree  variability. 

C.  On  the  Robustness  of  the  Bivariate  HOT  Model 

Our  bivariate  HOT  model  for  Internet  growth  at  the 
AS  pc  level  attempts  to  capture  the  local  decision  pro¬ 
cesses  performed  by  individual  ASs  within  an  abstract  but 
intuitive  framework.  In  accordance  to  the  model  valida¬ 
tion  framework  advocated  in  [32],  the  natural  next  step 
would  be  to  empirically  validate  the  model  using  rele- 
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Fig.  10.  Primary  vs.  secondary  criteria 


vant  data.  However,  at  this  point  it  befuddles  us  how 
to  collect  the  appropriate  data;  in  fact,  we  are  not  even 
sure  what  sort  of  data  might  be  available  that  could  shed 
some  light  on  the  validity  of  the  proposed  model,  espe¬ 
cially  as  far  as  the  assumed  toy  business  model  is  con¬ 
cerned.  Fully  realizing  this  dilemma,  the  next-best  ap¬ 
proach  in  support  of  the  overall  viability  of  the  model  is 
to  assess  its  sensitivity  to  changes  in  the  underlying  toy 
business  model  (aq,  Si)  associated  with  AS  i.  The  explo¬ 
ration  of  the  impact  of  such  changes  or  of  related  what- 
if  scenarios  is  typically  motivated  by  real-world  inter-AS 
peering  considerations  and  inevitably  complicates  the  re¬ 
sulting  business  model  beyond  our  intuitive  but  naive  toy 
model,  where — in  the  absence  of  empirical  evidence  to  the 
contrary — the  assignments  of  score  vectors  and  selection 
sets  are  completely  left  to  chance  (i.e.,  independent  and 
uniformly  random).  For  the  viability  of  our  proposed  bi¬ 
variate  model,  it  is  therefore  important  to  understand  what 
sort  of  changes,  refinements,  or  modifications  to  our  toy 
business  model  result  in  qualitatively  the  same  graph  struc¬ 
tures,  and  which  disturbances  give  rise  to  qualitatively  very 
different  topologies. 

Primary  vs.  secondary  criteria  To  illustrate,  the  fol¬ 
lowing  refinement  of  the  bivariate  HOT  model  is  motivated 
by  the  observation  that  when  evaluating  different  provider 
ASs,  customer  AS  i  may  be  more  concerned  with  some 
criteria  (e.g.,  unit  bandwidth  cost)  than  others  (e.g.,  cus¬ 
tomer  support).  This  then  suggests  considering  selection 
sets  Si  for  AS  i  that  are  not  necessarily  randomly  cho¬ 
sen  from  2^1,2,  ',Ar^  \  {(f)}.  To  achieve  this  goal,  we  as¬ 
sociate  with  a  given  node  i  an  ./V-tuple  of  weights  A(i)  = 
Si, 2,  Si, n),  0  <  Si,n  <  1,  where  Shn  is  the  weight 

with  which  the  node  i  indicates  its  preference  to  have 
criterion  n  included  in  its  selection  set.  We  then  define 
the  probability  that  the  node’s  selection  set  St  is  instanti¬ 
ated  with  a  particular  subset  X  in  2{1,2’-,Ar}  \  {(f>}  to  be 
((rLe.Y  Si,u){Ylv^tx  (1  —  Si, „)))/(!  —  Uu,=i  (-*-  —  Si,w)). 


By  varying  the  weight- vector  A(i),  we  are  able  to  explic¬ 
itly  control  the  relative  preference  for  including  certain  cri¬ 
teria  (or  indices  of  the  score  vector)  in  a  selection  set  over 
others.  For  example,  setting  all  weights  equal  to  0.5  cor¬ 
responds  to  the  purely  random  choice  of  selection  sets  as¬ 
sumed  in  our  original  formulation  of  the  bivariate  model. 
On  the  other  hand,  having  Sun  larger  than  5i,m  gives  rise 
to  selection  sets  that  are  more  likely  to  contain  criterion 
n  than  m.  Given  a  weight-vector  A(i),  we  next  classify 
the  score  vector  indices  or  criteria  (but  not  the  actual  val¬ 
ues  of  the  components  of  the  score  vector)  into  primary 
and  secondary  indices,  depending  on  whether  5i,n  >  0.5 
or  5i,n  <  0.5,  resp.  That  is,  a  primary  index  represents  a 
criteria  that  an  AS  deems  important  when  choosing  its  up¬ 
stream  provider;  secondary  indices  correspond  to  criteria 
that  the  AS  considers  to  be  relatively  unimportant. 

In  our  experiment,  we  tested  a  special  case  where  A(i) 
=  A(j)  =  A  for  every  pair  of  node  i  and  j  (i.e.,  the  same 
weight-vector  A  for  every  node),  and  all  primary  indices 
are  assigned  equally  high  weights  (i.e.,  0.8)  and  all  sec¬ 
ondary  indices  equally  low  weights  (i.e.,  0.2).  The  number 
of  primary  and  secondary  indices  in  A  is  then  denoted  by 
Np  and  Ns  respectively,  where  Np  +  Ns  =  N.  Under  this 
setting,  we  generated  a  set  of  graphs  varying  Np  from  1 
to  5  and  Ns  from  0  to  5.  Fig.  10(a)  shows  the  inferred 
a-values  (i.e.,  estimated  tail-index  of  power-law  distribu¬ 
tions  of  the  form  P[X  >  x\  ~  x~a,  for  large  x)  of  the 
degree  distributions  of  these  generated  graphs,  where  the 
a-estimates  are  the  slopes  of  least  squares  fits  on  log-log 
scale).  Fig.  10(b)  plots  their  connectivity  concentration  ra¬ 
tios  (CCRs)  introduced  in  Section  III-B.  The  reported  val¬ 
ues  in  both  figures  are  averages  taken  over  ten  realizations 
that  were  generated  for  each  pair  of  (Np,  A(,)- values.  For 
comparison,  we  added  to  the  figures  the  inferred  a-  and 
CCR-values  of  the  PI  subgraph  of  OREGON  AS  graph  (la¬ 
beled  “Internet”). 


17 


1 

0.6 

1 

1 

1 

_ 

_ + - 

x_. . " 

0.5 

_ -H - 

- 1- - 

. . . 

* . * 

i  cc 

- 

Internet  - 

. 

- 

o 

fa 

0.4 

n'  =  2  - 

-x  -  - 

- 

r'' 

X""' 

* 

,-Er' 

_ a" 

Cl) 

o 

c 

o 

0.3 

N[  =  5  - 

- 

x  . 

.  * 

" 

0.2  > 

. . 

N,  =  5  — -H- 

Nr  =  4  -  -x  --  - 

CD 

c 

c 

0.1  > 

. ' 

Nr  =  3  * 

Nr  =  2  — -0--~ 

Nr  =  1  ■ 

O 

_ E - 

i 

0 

( 

- ■ - 

0 

1 

2 

3 

4  * 

) 

) 

1 

2 

3 

4 

Nd  Nd 

(a)  Lineal-  fit  slope  (b)  Connectivity  concentration  ratio 


Fig.  11.  Regenerative  vs.  degenerative  criteria 


Fig.  10(a)  suggests  that  as  Np  or  Ns  increases,  the  slope 
of  the  degree  distribution  becomes  steeper.15  The  rea¬ 
son  for  this  behavior  is  similar  to  what  explains  Fig.  9(b). 
Due  to  different  selection  preferences  for  primary  and  sec¬ 
ondary  criteria,  however,  an  increase  of  Ns  has  much  less 
of  an  effect  on  the  degree  distribution  than  an  increase 
of  N.p.  In  addition.  Fig.  10(b)  shows  that  if  Np  >  1, 
the  CCR  of  a  generated  graphs  falls  short  of  that  of  the 
Internet  AS  graph.  From  this  experiment,  we  conclude 
that  when  allowing  for  a  separation  of  the  score  vector 
into  components  representing  primary  and  secondary  cri¬ 
teria,  Internet-like  node  degree  distributions  are  typically 
achieved  with  higher-dimensional  score  vectors  than  in  our 
original  model.  Furthermore,  to  retain  an  Internet-like  hi¬ 
erarchical  structure  (as  measured  in  terms  of  the  connec¬ 
tion  concentration  ratio  CCR),  the  value  of  Np  is  required 
to  be  smaller  than  the  dimension  of  the  score  vectors  in 
our  original  model. 

Regenerative  vs.  degenerative  criteria  Another  type 
of  refinement  to  our  toy  business  model  concerns  the  ac¬ 
tual  values  assigned  to  the  ASs’  score  vectors.  This  re¬ 
finement  reflects  an  aspect  of  networking  reality  whereby 
for  example  network  reliability  of  a  customer  AS  cannot 
be  higher  than  that  of  its  upstream  providers’,  mainly  be¬ 
cause  network  access  failures  inevitably  propagate  through 
downstream  customer  networks.  We  call  criteria  whose 
quality  for  the  customer  AS  is  provider  dependent  degen¬ 
erative  criteria.  In  contrast,  an  AS’s  quality  of  customer 
support  is  a  criterion  that  is  not  necessarily  determined  by 
its  upstream  providers’;  we  call  such  provider-independent 
criteria  regenerative  criteria.  The  introduction  of  degen¬ 
erative  criteria  leads  us  to  consider  the  impact  of  score 
vectors  that  are  not  necessarily  independent  across  differ¬ 
ent  ASs  and  whose  components  are  not  necessarily  cho¬ 
sen  uniformly  between  0  and  1.  In  our  original  bivariate 

15In  all  cases  reported  in  the  fi  gures,  the  quality  of  the  linear  fi  t  is 
very  high  with  correlation  coeffi  cient  >  0.98. 


model,  the  score  vector  x *  of  node  i  is  determined  inde¬ 
pendently  of  that  of  its  parent  node  j’s.  If  criterion  m  hap¬ 
pens  to  be  degenerative,  then  the  value  of  the  score  vector 
component  xt,m  is  chosen  uniformly  from  [0,  x:pm]  (i.e., 
it  depends  on  the  particular  AS  j  chosen  as  *’ s  provider 
and  on  j’s  score  vector).  Regenerative  criteria  are  assigned 
random  values  in  [0, 1]  as  in  the  original  bivariate  model, 
independent  of  everything  else.  The  number  of  degener¬ 
ative  and  regenerative  criteria  are  denoted  by  N,/  and  Nr 
respectively,  where  +  Nr  =  N. 

To  explore  the  parameter  space  associated  with  this  sec¬ 
ond  refinement  of  our  original  bivariate  model,  we  vary 
Nr  from  1  to  5  and  N(i  from  0  to  5  and  consider  the  case 
of  uniformly  random  selection  sets.  Fig.  1 1(a)  is  consistent 
with  our  earlier  observations  that  higher  dimensional  score 
vectors  result  in  steeper  degree  distributions.  In  addition, 
one  can  see  that  increasing  Nj  while  fixing  N  does  not 
change  the  slope  of  the  degree  distribution  significantly. 
For  example,  the  four  dots  corresponding  to  N  =  4  remain 
close  to  the  horizontal  reference  line.  Fig.  11(b)  indicates 
that  a  higher  Nt/  yields  a  graph  with  a  higher  CCR  value. 
The  effect  of  changing  Nrj  is  more  dramatic  when  N  is 
fixed.  For  example,  incrementing  Nfj  from  0  to  3  with 
N  =  4  causes  the  CCR  of  a  generated  graph  to  increase 
from  0.05  up  to  0.5.  With  more  degenerative  criteria,  cus¬ 
tomer  ASs  are  more  likely  connect  directly  to  nodes  closer 
to  the  root,  resulting  in  an  overall  topology  that  becomes 
more  star-shaped  from  the  root  node  outward.  The  main 
finding  from  this  set  of  experiments  is  that  the  presence  of 
degenerative  qualities  or  criteria  provides  a  direct  “dial” 
for  impacting  the  growth  dynamics  of  the  graphs’  cores  as 
measured  by  the  CCR  metric. 

Two  refi  nements  in  combination  Finally,  we  also  ex¬ 
perimented  with  combining  the  previous  two  refinements 
and  allowed  ASs  to  classify  their  criteria  to  be  primary  or 
secondary  and,  at  the  same  time,  degenerative  or  regener¬ 
ative.  Out  of  Np  primary  criteria,  Npfj  criteria  have  de- 
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Fig.  12.  Primary  regenerative  vs.  primary  degenerative  criteria 


generative  property  and  the  rest  of  Npr  criteria  have  re¬ 
generative  property  (Np<]  +  Npr  =  Np).  Likewise  with  Ns 
secondary  criteria  (Ns  =  Nsci  +  Nsr).  Fig.  12  shows  the  a- 
estimates  of  the  degree  distributions  and  the  CCR-values 
of  graphs  generated  with  Np  =  2  and  0  <  Ns  <5.  Pri¬ 
mary  criteria  are  provided  with  either  degenerative  or  re¬ 
generative  property,  but  all  secondary  criteria  are  classified 
as  regenerative  by  default  (i.e.,  Ns  =  Arsr). 16  Fig.  12(b) 
shows  that  if  all  primary  elements  have  degenerative  prop¬ 
erty  (i.e.,  (Npr,Np(])=(0,2))  and  Ns  is  small,  the  resulting 
graphs  tend  to  favor  a  pronounced  star-shaped  topology. 
On  the  other  hand,  if  all  primary  criteria  have  the  regener¬ 
ative  property  (i.e.,(Npr,Npci)={2,0)),  the  CCR  of  a  gener¬ 
ated  graphs  fall  short  of  that  of  the  Internet  AS  graph.  In 
short,  neither  of  the  two  extreme  cases  yields  the  Internet¬ 
like  hierarchical  structure  expressed  by  CCR.  As  long  as 
we  avoid  such  extreme  cases,  the  refined  model  results  in 
the  Internet-like  node  degree  distribution  and  hierarchical 
structure  (e.g.,  {Npr,Npd)={  1,1)  and  Ns  =  2).  We  made 
similar  observations  for  Np  >  2  as  well. 

The  main  conclusions  from  experimenting  with  these 
refinements,  either  individually  or  in  combination,  are  that 
the  qualitative  aspects  of  the  resulting  graphs  (as  captured 
for  example  by  the  node  degree  distribution  or  the  connec¬ 
tivity  concentration  ratio  CCR  introduced  in  Section  III- 
B)  by  and  large  agree  with  those  obtained  using  the  orig¬ 
inal  bivariate  model.  Quantitatively,  the  graphs  differ  in 
very  intuitive  and  predictable  ways  as  a  result  of  fine- 
tuning  their  growth  dynamics  through  particular  choices 
of  the  parameters  associated  with  each  refinement;  that 
is,  the  number  of  primary  and  secondary,  or  degenerative 
and  regenerative  criteria.  Note  that  these  parameters  pro¬ 
vide  explicit  “knobs”  for  manipulating  the  sizing  of  the 

15We  also  tested  the  cases  where  some  secondary  criteria  have  degen¬ 
erative  property.  However,  as  far  as  secondary  criteria  are  concerned, 
whether  they  are  degenerative  or  regenerative  does  not  make  much  dif¬ 
ference. 


AS-specific  sets  of  Pareto-optimal  candidate  nodes,  which 
in  turn  determines  node  degree  variability  and  hierarchi¬ 
cal  structure  of  the  resulting  graphs.  For  example,  when 
allowing  for  degenerative  and  regenerative  criteria,  with 
more  degenerative  criteria,  customer  ASs  are  more  likely 
to  connect  directly  to  nodes  closer  to  the  root,  resulting  in 
an  overall  topology  that  becomes  more  star-shaped,  with 
the  root  at  the  center.  On  the  other  hand,  when  ASs  arc 
given  the  choice  to  classify  their  criteria  to  be  regenera¬ 
tive  or  degenerative  and,  at  the  same  time,  primary  or  sec¬ 
ondary,  we  find  that  when  customer  ASs  select  among  a 
number  of  competing  provider  ASs,  neither  the  primary 
regenerative  nor  the  primary  degenerative  criteria  of  the 
providers  dominate  the  local  decision  processes  of  cus¬ 
tomer  ASs,  but  instead,  both  types  of  criteria  are  actively 
considered  in  their  decision-making  processes. 

VI.  AS  Graph  Evolution  as  a  Causal  Force 

The  third  and  final  step  of  our  construction  of  a  new 
HOT  model  for  Internet  growth  at  the  ASpc  level  con¬ 
cerns  primarily  the  graph  growth  process  itself.  This  step 
is  motivated  by  the  observation  that  while  the  bivariate  (as 
well  as  the  univariate)  model  is  purely  incremental  in  na¬ 
ture  (i.e.,  nodes  are  added  to  the  graph  one  by  one,  and 
once  added,  they  stay  forever,  and  so  do  all  the  added 
links),  the  historical  evolution  of  AS-related  connectivity 
is  obviously  more  dynamic,  given  the  business  dynamics 
of  the  ISP  market. 

A.  The  Multivariate  HOT  Model 

In  incrementally  grown  graphs  like  the  ones  generated 
by  our  bivariate  HOT  model,  the  parent  (or  ancestor)  node 
of  any  given  node  i  is  always  added  to  the  graph  before 
node  i.  In  the  Internet,  however,  existing  ASs  can  and  do 
disappear  as  a  result  of,  for  example,  bankruptcies,  often 
times  leaving  their  customers  scrambling  for  new  provider 
ASs.  To  account  for  such  change-of-provider  scenarios 
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in  our  model,  we  consider  the  bivariate  HOT  model  de¬ 
fined  in  Section  V-B  and  introduce  node  death  events  that 
trigger  a  particular  change- of -provider  mechanism.  More 
precisely,  every  time  Pt\  new  nodes  have  been  added  to 
the  graph,  one  of  the  existing  nodes  u  in  the  graph  is  ran¬ 
domly  selected  and  removed  from  the  graph  (together  with 
all  its  incident  links).  In  turn,  all  of  node  it’s  children  (if 
any)  become  orphans  and  have  to  select  a  new  parent  node 
to  re-establish  graph  connectivity.  When  determining  its 
new  provider,  the  local  decision  process  of  a  given  orphan 
node  i  is  similar  to  the  one  by  which  a  newly  arriving 
node  connects  to  the  graph  in  our  bivariate  HOT  model, 
but  differs  in  two  important  ways.  First,  when  an  orphan 
node  i  chooses  its  set  of  “close-by”  candidate  providers, 
none  of  the  potential  new  parent  nodes  can  be  descendants 
of  node  i.  Second,  the  definition  of  geographic  proxim¬ 
ity  needs  to  be  modified  because  in  contrast  to  a  newly 
arriving  node  u  that  is  assigned  a  locJist(u )  containing 
a  single  PoP  location,  the  orphan  node  i’ s  internal  PoP 
structure  as  described  in  locJist(i )  may  have  grown  sub¬ 
stantially  since  node  V s  birth.  To  account  for  this  lat¬ 
ter  complication,  we  define  the  geographic  proximity  be¬ 
tween  an  orphan  node  i  and  a  potential  new  parent  node  j 
to  be  (EfeetocJiSt(i)(™nt6/ocJiSt(j)4t))/|^cJfsf(i)|,  the 
expected  minimum  distance  between  nodes  i  and  j.  If 
the  orphan  node  i  happens  to  have  only  a  single  loca¬ 
tion,  this  expression  reduces  to  mini&ioc  ust(j)du,  the  orig¬ 
inal  definition  of  geographic  proximity  used  in  our  uni¬ 
variate/bivariate  model.  We  call  this  final  HOT  model 
with  its  multi-faceted  objectives  (i.e.,  AS  geography, 
AS  business  model,  and  AS  evolution)  the  multivariate 
model  for  Internet  growth  at  the  AS  pc  level. 

To  judge  the  impact  of  allowing  for  evolutionary 
changes  through  the  introduction  of  node  death  events  and 
its  associated  change-of-provider  mechanism,  we  gener¬ 
ated  two  graphs  (each  with  10,000  nodes).17  The  first 
graph  was  generated  using  our  bivariate  HOT  model  (with 
K  =  0.05,  j3  =  1.0,  and  N  =  3);  the  second  graph 
resulted  from  an  identically  parameterized  version  of  our 
multivariate  HOT  model,  where  in  addition,  we  set  Pd  = 
5. 18  While  the  node  degree  distributions  of  the  resulting 
graphs  arc  practically  identical  (not  shown),  the  scatter- 
plots  of  node  age  vs.  node  degree  depicted  in  Fig.  13  illus¬ 
trate  the  impact  that  the  introduction  of  change-of-provider 
events  has  on  the  graph  growth  process  itself.  We  observe 

1 '  To  generate  a  graph  with  N  fi  nal  nodes  using  the  multivariate 
model,  we  need  to  add  N  •  pyrj  n°des,  out  of  which  p^_1  nodes 
will  be  removed  during  the  growth  process. 

18The  chosen  value  of  P,j  is  an  approximate  ratio  of  AS  birth  fre¬ 
quency  and  AS  death  frequency  that  are  empirically  observed  from  the 
Oregon  data  sets  described  in  Section  VI-B. 


that  by  allowing  for  these  specified  evolutionary  changes, 
the  multivariate  model  in  plot  (b)  introduces,  as  expected, 
more  randomness  between  node  ages  and  node  degrees 
than  the  bivariate  model  in  plot  (a),  thereby  reducing  the 
correlation  between  node  age  and  degree  to  be  more  con¬ 
sistent  with  that  observed  in  inferred  AS  pc  graphs.  Note 
that  in  the  case  of  these  generated  graphs,  node  age  was 
calculated  based  on  an  empirically  derived  AS  birth  rate 
(see  appendix  for  details). 

B.  On  the  Historical  AS  Evolution 

Unfortunately,  inference  for  historical  AS  evolution 
(i.e.,  AS  birth,  AS  death,  change-of-provider,  becoming 
a  peer  or  customer  AS,  etc.)  is  compromised  by  the  ab¬ 
sence  of  reliable  AS  meta-data.  This  difficulty  not  with¬ 
standing,  we  rely  on  a  hand-crafted  approach  for  obtaining 
qualitative  rather  than  quantitative  evidence  in  support  of 
including  AS  evolution  into  our  multivariate  HOT  model. 
In  particular,  relying  on  the  daily  data  sets  from  the  Oregon 
route  server  which  span  the  period  Nov.  1997  to  May  2001 
and  applying  a  previously  reported  methodology  for  iden¬ 
tifying  actual  AS  births  (see  [12]),  we  carefully  extracted 
a  set  of  ASs  that  were  born  during  this  period.19  For  those 
newly  born  ASs  that  were  still  alive  at  the  end  of  our  data 
collection  period  (May  26,  2001),  we  calculated  their  ages 
with  respect  to  May  26,  2001.  For  the  ASpc  subgraph  in¬ 
ferred  from  the  Internet  AS  topology  of  May  26,  2001,  we 
obtained  the  actual  ages  of  8,799  ASs.  The  age  difference 
between  provider-customer  pairs  when  both  ASs  belong  to 
these  8,799  ASs  can  be  easily  computed.  For  ASs  born  be¬ 
fore  the  start  of  Oregon’s  data  collection  effort  (Nov.  1997) 
and  whose  exact  age  information  is  therefore  not  available, 
we  set  their  ages  to  the  maximum  age  of  the  8,799  ASs. 

Given  an  AS  X,  if  all  the  current  parent  ASs  of  X  are 
younger  than  X,  X  is  considered  to  have  undergone  a 
change-of-provider.  By  this  criterion,  almost  3%  of  the 
ASs  present  in  the  May  26,  2001  provider-customer  sub¬ 
graph  have  changed  provider  at  least  once  during  their  life¬ 
time.  With  a  less  stringent  definition  of  what  constitutes  a 
change-of-provider  (i.e.,  a  fraction  of  AS  X’s  parent  ASs 
are  younger  than  X),  some  15%  of  all  the  ASs  present  in 
the  May  26,  2001  subgraph  may  have  undergone  a  change 
of  providers.  As  to  age-degree  correlation,  plot  (c)  in 
Fig.  13  shows  the  scatter-plot  for  those  8,799  AS  for  which 
we  have  actual  age  information  and  which  are  paid  of  the 

19We  also  experimented  with  using  AS  number  as  a  substitute  for  AS 
age,  but  found  that  due  to  AS  number  recycling  and  the  specifi  cs  of  AS 
number  allocation  (e.g.,  disjoint  AS  number  ranges  are  allocated  to  the 
different  Regional  Internet  Registries,  which  in  turn  assigned  individual 
AS  numbers  sequentially),  inferring  AS  age  form  AS  number  may  be 
too  crude  of  an  approximation. 
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(single-homed)  provider-customer  subgraphs  of  the  Ore- 
GON+  AS  graph.  As  can  be  seen,  there  is  no  pronounced 
correlations  between  the  two  quantities;  many  of  the  older 
ASs  have  degree  one  or  two,  and  the  age  of  those  ASs  with 
degree  bigger  than,  say,  10  ranges  widely  across  the  entire 
a;- axis.  While  this  plot  agrees  qualitatively  with  that  of 
our  multivariate  HOT  model  in  plot  (b) — in  that  both  sug¬ 
gest  little  correlation  between  AS  age  and  AS  node  degree, 
they  also  show  a  discrepancy  in  the  range  of  node  degrees. 
We  conjecture  that  the  absence  of  100+  degree  nodes  in 
plot  (c)  is  due  to  the  fact  that  in  the  real  Internet,  high- 
degree  customer  ASs  will  attempt  to  become  peers  of  their 
providers.  This  aspect  of  AS  evolution  has  been  explicitly 
ignored  by  our  deliberate  focus  on  the  provider-customer 
portion  of  the  AS  graph,  but  arises  at  this  point  as  a  natural 
next  step  in  the  development  of  a  genuine  HOT  model  for 
Internet  growth  at  the  AS  level  as  a  whole.  The  develop¬ 
ment  of  such  a  more  complete  HOT  model  that  also  ac¬ 
counts  for  peer  ASs  (i.e.,  their  historical  evolution  and  ap¬ 
propriate  business  models)  and  peer-to-peer  relationships 
is  beyond  the  scope  of  this  paper  and  is  left  as  future  work. 

VII.  Discussion  and  Outlook 

One  of  the  features  of  the  proposed  multivariate  HOT 
model  for  Internet  growth  at  the  ASpc  level  is  that  by 
explaining  the  striking  power-law  type  node  degree  dis¬ 
tributions  of  inferred  AS  graphs  and  subgraphs  in  terms 
of  power-law  type  distributions  for  inferred  AS  size  (i.e., 
the  number  of  PoPs  per  AS),  it  confirms  a  previously  re¬ 
ported  conjecture  in  [31],  namely  that  “the  highly  vari¬ 
able  degree  distribution  may  arise  merely  from  its  corre¬ 
lation  with  a  highly  variable  size  distribution Naturally, 
such  an  explanation  can  be  rightly  criticized  as  answer¬ 
ing  one  question  (i.e.,  what  causes  power-law  type  AS 
degree  distributions?)  with  another  question  (i.e.,  what 
causes  power-law  type  AS  size  distribution?)  To  respond 
to  this  criticism,  first  note  that  our  explanation  has  shifted 
the  focus  from  an  abstract  concept  (i.e.,  node  degree)  to 


a  concrete  object  (i.e.,  AS).  Moreover,  by  viewing  ASs  as 
genuine  businesses  or  firms,  we  can  now  bring  to  hear  an 
extensive  existing  literature  on  empirical  studies  of  firm 
size  distributions  that  provide  compelling  evidence  of  the 
ubiquity  of  power-law  type  distributions  of  firm  sizes,  ir¬ 
respective  of  how  firm  size  is  measured  (e.g.,  number  of 
employees,  revenues,  sales  volume,  customer  base,  num¬ 
ber  branch  offices  or  outlets).  In  this  sense,  the  situation 
is  comparable  to  explaining  the  high  variability  or  power- 
law  type  distributions  exhibited  by  the  constituent  compo¬ 
nents  of  aggregate  network  traffic  (e.g.,  sizes  or  durations 
of  TCP  connections,  IP  flows,  macro-flows,  etc.)  in  terms 
of  power-law  type  distributions  for  the  sizes  of  files  on  file 
servers  or  the  sizes  of  Web  documents  on  Web  servers. 
The  latter  have  proven  to  be  a  broadly  robust  properties 
over  time  and  across  different  servers,  despite  often  drastic 
changes  in  user  behavior  or  application  design.  Similarly, 
firm  sizes,  measured  variously,  have  been  shown  to  consis¬ 
tently  exhibit  power-law  type  behavior,  over  many  years 
and  decades  and  for  different  nations,  despite  constant 
merger  and  acquisition  activities,  as  well  as  bankruptcies 
(see  for  example  [5],  [27]  and  references  therein).  Thus 
our  approach  is  to  take  the  high  variability  of  firm  sizes 
for  granted,  just  as  we  have  come  to  accept  the  power-law 
type  distributions  for  file  or  Web  document  sizes.  How¬ 
ever,  we  also  want  to  point  out  that  studies  that  attempt  to 
explain  the  power-law  type  behavior  of  file  sizes  or  firm 
sizes  are  of  interest  in  their  own  right  and  have  attracted 
considerable  attention  (e.g.,  [33],  [14],  [25],  [4]). 

A  major  contribution  of  this  paper  is  make  the  AS  busi¬ 
ness  model  concept  a  vital  part  of  the  multivariate  HOT 
model  for  Internet  growth  at  the  ASpc  level.  By  abstract¬ 
ing  the  vague  concept  to  an  intuitively  appealing  but  ad¬ 
mittedly  naive  mathematical  toy  model,  we  demonstrated 
its  potential  for  exploring  some  simple  what-if  scenarios 
through  a  systematic  investigation  of  the  model’s  low¬ 
dimensional  parameterization.  For  example,  our  findings 
in  Section  V-C  suggest  that  in  today’s  Internet,  the  eco- 
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nomics  of  establishing  provider-customer  relationships  is 
not  very  complex  and  appeal's  to  be  based  on  a  relatively 
small  number  of  basic  criteria  and  objectives.  In  turn,  our 
toy  AS  business  models  also  predict  that  if  an  ISP’s  pri¬ 
mary  task  in  the  future  is  no  longer  simply  to  amass  ad¬ 
equate  resources  to  build  out  its  infrastructure  to  fuel  the 
Internet’s  overall  growth,  but  becomes  more  sophisticated 
due  to  emerging  refinements  of  interconnection  business 
structures,  we  can  expect  to  see  qualitative  changes  in  the 
resulting  AS  graph  structures.  However,  a  careful  inves¬ 
tigation  of  the  full  potential  of  the  business  model  con¬ 
cept,  including  more  sophisticated,  elegant,  and  versatile 
abstractions  is  left  for  future  research. 

We  emphasize  that  while  the  proposed  multivariate 
HOT  model  has  been  designed  to  explicitly  describe  In¬ 
ternet  growth  at  the  ASpc  level  and  not  at  the  overall 
AS  level  (i.e.,  including  peer-to-peer  relationships),  it  is 
nevertheless  capable  of  explaining  observed  phenomena  in 
inferred  (overall)  AS  graphs,  provided  the  essential  char¬ 
acteristics  of  these  phenomena  are  already  present  in  the 
AS  pc  subgraphs.  The  high  variability  of  AS  node  degrees 
is  one  such  characteristic.  To  develop  a  HOT  model  for 
Internet  growth  that  applies  to  the  entire  AS  graph  and  not 
just  to  the  portion  corresponding  of  the  ASpc  subgraph 
would  require  an  appropriate  treatment  of  peer-to-peer  re¬ 
lationships,  either  within  the  framework  of  our  current  AS 
business  model  or  by  modifying  the  present  model  to  ex¬ 
plicitly  account  for  the  possibilities  of  customer  ASs  be¬ 
coming  peers  and  peers  turning  into  customers  (see  for 
example  our  conjecture  in  Section  VI).  In  either  case, 
the  resulting  HOT  model  can  be  expected  to  lend  itself  to 
easy  generation  of  realistic  topologies  at  the  AS  level,  with 
the  appealing  property  that  Internet-like  AS  connectivity  is 
obtained  and  guaranteed  by  imitating  the  very  distributed 
and  decentralized  approach  that  underlies  the  design  of  AS 
connectivity  in  the  actual  Internet.  All  the  model  param¬ 
eters  have  physical  meaning,  and  using  them  as  “knobs” 
results  in  predictable  and  intuitively  easy-to-grasp  refine¬ 
ments  of  the  resulting  graphs’  overall  structures. 

Finally,  within  the  broader  context  of  Internet  model¬ 
ing  as  a  science  and  in  light  of  an  extensive  literature  that 
advocates  the  orthodox  physics  view  that  power-law  type 
behavior  is  unambiguously  related  to  critical  phase  transi¬ 
tion  [6],  our  approach  suggests  a  simple  recipe  for  separat¬ 
ing  sound  from  specious  claims  and  theories:  use  domain 
knowledge  and  check  against  appropriate  measurements. 
For  example,  when  comparing  the  scale-free  models  for 
Internet  growth  at  the  AS  level  introduced  in  [8],  [2]  with 
our  multivariate  HOT  model,  we  find  that  the  former  is 
void  of  any  domain  knowledge  and  can  be  easily  refuted 
using  available  measurements  about  the  network’s  histor¬ 


ical  evolution  [12],  [32].  In  contrast,  the  latter  not  only 
thrives  on  domain  knowledge  and  incorporates  it  explic¬ 
itly  into  the  model  formulation,  but  is  also  fully  consistent 
with  a  number  of  available  measurements  that  provide  rel¬ 
evant  information  about  all  different  aspects  of  the  model. 
Given  such  an  attractive  alternative,  it  will  be  difficult  to 
argue  for  the  networking  relevance  of  models  and  theories 
that  can  be  easily  refuted,  both  analytically  as  well  as  em¬ 
pirically. 
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Appendix 

I.  On  Inferring  AS  Birth  Rate 

Fig.  14(a)  and  (b)  plot  daily  AS  birth  rate  (i.e.,  the  num¬ 
ber  of  ASs  born  per  day)  and  the  total  number  of  ASs  born 
since  Nov.  1997,  respectively.  According  to  Fig.  14(a), 
AS  birth  rate  has  remained  flat  till  day  600  or  so,  after 
which  it  started  to  ramp  up  till  around  day  800,  and  has 
been  stable  ever  since.  Fig.  14(b)  confirms  this  AS  birth 
rate  transition.  That  is,  the  “total  number  of  births”  curve 
in  Fig.  14(b)  can  be  approximated  by  concatenating  two 
linear  lines  that  meet  around  day  700.  Based  on  this  ob¬ 
servation,  we  characterize  historical  AS  birth  rate  by  two 
distinctly  constant  values,  which  is 


birth-rate(t) 


C i ,  if  0  <  t.  <  To; 
C2,  if  t  >  Tq, 


(1) 


where  t  =  0  is  the  time  when  the  Internet  AS  graph 
came  into  being,  t  =  Tq  is  the  time  when  the  transition  of 


AS  birth  rate  occurred,  and  Cf  and  C2  are  positive  con¬ 
stants  with  Ci  <  C220 

Calculating  the  slopes  of  the  curve  of  Fig.  14(b)  in  two 
separate  time  periods  [0,700]  and  [700,1200]  yields  C\  = 
5.4274  and  CP  =  1 3. 1 .  In  order  to  approximately  determine 
To,  we  note  that: 

#  of  ASs  at  To  =  #  of  AS  births  during  [0,To]  -  #  of  AS 
deaths  during  [0,To]  =  ( To, 

-'a 

where  P,i  is  the  ratio  of  the  number  of  AS  births  and  the 
number  of  AS  deaths  during  [0,To],  However,  since  the  AS 
birth/death  history  data  is  available  only  from  Nov.  1997, 
we  approximate  Pr]  with  Pfj,  which  is  the  ratio  of  the  num¬ 
ber  of  AS  births  and  the  number  of  AS  deaths  during 
[Nov.  1997, To].  This  approximation  yields  To  =  1,390, 
which  translates  the  birth  time  of  the  Internet  to  Jan.  1996. 

If  the  node  birth  rate  as  shown  in  (1)  is  assumed  in  our 
multivariate  model,  the  age  of  node  i  in  a  graph  where  the 
maximum  node  id  is  N  can  be  characterized  as: 


age(i)  = 


if  i  <  CiTo; 
if  i  >  C  i  To . 


(2) 

Node  ages  calculated  by  the  above  formula  have  the 
same  time  unit  as  in  the  Internet  AS  graph  (i.e.,  the  num¬ 
ber  of  days),  and  therefore  node  ages  and  AS  ages  can  be 
directly  compared. 


20Note  that  the  available  historical  AS  data  sets  do  not  date  back  to 
t  =  0.  The  start  date  (Nov.  1997)  of  the  date  sets  lies  between  t=0  and 
i=To-  For  the  purpose  of  this  paper,  we  assume  that  AS  birth  rate  for 
the  period  [0,  Nov.  1997]  has  remained  the  same  as  that  for  the  period 
[Nov.  1997,  T0], 


Number  of  AS  Bi 


